[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:43:58 dist-test-slave-j968 krb5kdc[13340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:43:58 dist-test-slave-j968 krb5kdc[13340](info): set up 2 sockets
Sep 28 15:43:58 dist-test-slave-j968 krb5kdc[13340](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 15:43:59.819319 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:43:59 dist-test-slave-j968 krb5kdc[13340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:00.001207 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:41251
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:32877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:00.333060 13353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:00.333384 13353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:00.333572 13353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:00.341024 13353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:00.341138 13353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:00.341246 13353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:00.341364 13353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:00.341452 13353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:00.351630 13353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:32877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:41251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:00.353272 13353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:00.354732 13353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:44:00.368315 13353 server_base.cc:1034] running on GCE node
W20260928 15:44:00.368353 13359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:00.368287 13361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:00.368746 13358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:00.369688 13353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:00.371304 13353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:00.372526 13353 hybrid_clock.cc:648] HybridClock initialized: now 1790610240372455 us; error 56 us; skew 500 ppm
I20260928 15:44:00.372884 13353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:00 dist-test-slave-j968 krb5kdc[13340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610240, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:00.376260 13353 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:00.377936 13353 webserver.cc:460] Webserver started at http://127.13.5.126:36065/ using document root <none> and password file <none>
I20260928 15:44:00.378710 13353 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:00.378841 13353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:00.379176 13353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:00.382323 13353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "11cfb177b4044471b6c01b5773ec9af3"
format_stamp: "Formatted at 2026-09-28 15:44:00 on dist-test-slave-j968"
server_key: "7db3918976283bb8b2aea1b9111aae0b"
server_key_iv: "7489a9cc8567efed228fda60b4d725e9"
server_key_version: "encryptionkey@0"
I20260928 15:44:00.383095 13353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "11cfb177b4044471b6c01b5773ec9af3"
format_stamp: "Formatted at 2026-09-28 15:44:00 on dist-test-slave-j968"
server_key: "7db3918976283bb8b2aea1b9111aae0b"
server_key_iv: "7489a9cc8567efed228fda60b4d725e9"
server_key_version: "encryptionkey@0"
I20260928 15:44:00.388553 13353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:44:00.392194 13368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:00.393541 13353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:00.393735 13353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "11cfb177b4044471b6c01b5773ec9af3"
format_stamp: "Formatted at 2026-09-28 15:44:00 on dist-test-slave-j968"
server_key: "7db3918976283bb8b2aea1b9111aae0b"
server_key_iv: "7489a9cc8567efed228fda60b4d725e9"
server_key_version: "encryptionkey@0"
I20260928 15:44:00.393939 13353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:00.420766 13353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:00.425199 13353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:00.425552 13353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:00.445042 13353 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:41251
I20260928 15:44:00.445093 13419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:41251 every 8 connection(s)
I20260928 15:44:00.446539 13353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:00.448390 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 13353
I20260928 15:44:00.448736 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:00.451349 13333 external_mini_cluster.cc:1428] Setting key 5799bba35c02119298848b933b308421
I20260928 15:44:00.452031 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:00.464303 13420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3: Bootstrap starting.
I20260928 15:44:00.468389 13420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:00.469774 13420 log.cc:824] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:00.473332 13420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3: No bootstrap required, opened a new log
I20260928 15:44:00.482254 13420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cfb177b4044471b6c01b5773ec9af3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41251 } }
I20260928 15:44:00.482758 13420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:00.482883 13420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11cfb177b4044471b6c01b5773ec9af3, State: Initialized, Role: FOLLOWER
Sep 28 15:44:00 dist-test-slave-j968 krb5kdc[13340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:00.483624 13420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cfb177b4044471b6c01b5773ec9af3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41251 } }
I20260928 15:44:00.483866 13420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:00.484069 13420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:00.484251 13420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:00.487916 13420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cfb177b4044471b6c01b5773ec9af3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41251 } }
I20260928 15:44:00.488608 13420 leader_election.cc:302] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11cfb177b4044471b6c01b5773ec9af3; no voters:
I20260928 15:44:00.489392 13420 leader_election.cc:288] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:00.489509 13425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:00.490787 13425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [term 1 LEADER]: Becoming Leader. State: Replica: 11cfb177b4044471b6c01b5773ec9af3, State: Running, Role: LEADER
I20260928 15:44:00.491428 13425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cfb177b4044471b6c01b5773ec9af3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41251 } }
I20260928 15:44:00.492036 13420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:00.494704 13422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:00.456502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.13.5.126:41251)
0928 15:44:00.457595 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:00.457621 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:00.459217 (+ 1596us) server_negotiation.cc:407] Connection header received
0928 15:44:00.468649 (+ 9432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:00.468675 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:00.469223 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:00.469675 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:00.472914 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:00.475538 (+ 2624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:00.477373 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:00.478442 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:00.478659 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:00.484594 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:00.484618 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:00.484641 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:00.484699 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:00.488753 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:00.489677 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:00.489693 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:00.489699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:00.489980 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:00.490617 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:00.490622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:00.490625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:00.491335 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:00.491685 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:00.492777 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0928 15:44:00.493255 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":743,"thread_start_us":441,"threads_started":1}
I20260928 15:44:00.496029 13426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11cfb177b4044471b6c01b5773ec9af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cfb177b4044471b6c01b5773ec9af3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41251 } } }
I20260928 15:44:00.496469 13426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:00.497109 13427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11cfb177b4044471b6c01b5773ec9af3. Latest consensus state: current_term: 1 leader_uuid: "11cfb177b4044471b6c01b5773ec9af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cfb177b4044471b6c01b5773ec9af3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41251 } } }
I20260928 15:44:00.497447 13427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:00.502722 13431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:00.508925 13431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:00.518347 13431 catalog_manager.cc:1347] Generated new cluster ID: f36b9e8bdc784ddba1e1e8c1fbe83f4d
I20260928 15:44:00.518458 13431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:00.538455 13431 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:00.540033 13431 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:00.553896 13431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11cfb177b4044471b6c01b5773ec9af3: Generated new TSK 0
I20260928 15:44:00.554850 13431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:00.608378 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41251
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:32877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:00.961886 13447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:00.962168 13447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:00.962332 13447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:00.969780 13447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:00.969897 13447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:00.970084 13447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:00.980074 13447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:32877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:00.981401 13447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:00.982729 13447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:00.996553 13452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:00.996594 13455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:00.996515 13453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:00.997166 13447 server_base.cc:1034] running on GCE node
I20260928 15:44:00.997973 13447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:00.999024 13447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:01.000231 13447 hybrid_clock.cc:648] HybridClock initialized: now 1790610241000172 us; error 50 us; skew 500 ppm
I20260928 15:44:01.000579 13447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:00 dist-test-slave-j968 krb5kdc[13340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610240, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:01.004366 13447 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:01.006001 13447 webserver.cc:460] Webserver started at http://127.13.5.65:34429/ using document root <none> and password file <none>
I20260928 15:44:01.006747 13447 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:01.006891 13447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:01.007230 13447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:01.010699 13447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "a2255df589c9485dbb0f964f8be3d0c6"
format_stamp: "Formatted at 2026-09-28 15:44:01 on dist-test-slave-j968"
server_key: "bded54dcb27d9c0eab50b14f74b6a3b9"
server_key_iv: "2bfdfc01f9aadebf8f2814da2328a4c6"
server_key_version: "encryptionkey@0"
I20260928 15:44:01.011610 13447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "a2255df589c9485dbb0f964f8be3d0c6"
format_stamp: "Formatted at 2026-09-28 15:44:01 on dist-test-slave-j968"
server_key: "bded54dcb27d9c0eab50b14f74b6a3b9"
server_key_iv: "2bfdfc01f9aadebf8f2814da2328a4c6"
server_key_version: "encryptionkey@0"
I20260928 15:44:01.016906 13447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:44:01.020396 13462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:01.021792 13447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:44:01.021967 13447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "a2255df589c9485dbb0f964f8be3d0c6"
format_stamp: "Formatted at 2026-09-28 15:44:01 on dist-test-slave-j968"
server_key: "bded54dcb27d9c0eab50b14f74b6a3b9"
server_key_iv: "2bfdfc01f9aadebf8f2814da2328a4c6"
server_key_version: "encryptionkey@0"
I20260928 15:44:01.022163 13447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:01.036804 13447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:01.040939 13447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:01.041240 13447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:01.042255 13447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:01.043619 13447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:01.043720 13447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:01.043889 13447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:01.043967 13447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 15:44:01.077628 13447 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:42369
I20260928 15:44:01.077666 13574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:42369 every 8 connection(s)
I20260928 15:44:01.078933 13447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:01.087688 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 13447
I20260928 15:44:01.087960 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:01.088750 13333 external_mini_cluster.cc:1428] Setting key 97c77ef69857b624817a9b655e9c8993
Sep 28 15:44:01 dist-test-slave-j968 krb5kdc[13340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610240, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:01.100061 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:01.081702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:50615 (local address 127.13.5.126:41251)
0928 15:44:01.082142 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:01.082146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:01.083144 (+ 998us) server_negotiation.cc:407] Connection header received
0928 15:44:01.085017 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:01.085021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:01.085088 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:01.085217 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:01.086648 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:01.087478 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:01.089796 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:01.090225 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:01.090404 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:01.094245 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:01.094250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:01.094253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:01.094285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:01.096858 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:01.097621 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:01.097630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:01.097632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:01.097757 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:01.098246 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:01.098249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:01.098251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:01.098518 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:01.098696 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:01.099409 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 15:44:01.099647 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":206,"threads_started":1}
I20260928 15:44:01.101085 13578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:01.082260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:41251 (local address 127.13.5.65:50615)
0928 15:44:01.082963 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:01.083020 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:01.084753 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:01.085411 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:01.085418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:01.085900 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:01.086467 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:01.086483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:01.087653 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:01.087661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:01.089618 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:01.089630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:01.090577 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:01.090587 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:01.090818 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:01.091563 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:01.091585 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:01.094016 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:01.097021 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:01.097030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:01.097047 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:01.097489 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:01.097870 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:01.097875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:01.097879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:01.098102 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:01.098647 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:01.098655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:01.099013 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 15:44:01.099301 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 15:44:01.099856 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":272,"threads_started":1}
I20260928 15:44:01.102778 13575 heartbeater.cc:347] Connected to a master server at 127.13.5.126:41251
I20260928 15:44:01.103157 13575 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:44:01.104043 13575 heartbeater.cc:511] Master 127.13.5.126:41251 requested a full tablet report, sending...
I20260928 15:44:01.106355 13385 ts_manager.cc:194] Registered new tserver with Master: a2255df589c9485dbb0f964f8be3d0c6 (127.13.5.65:42369)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
I20260928 15:44:01.107982 13385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:50615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:01.127977 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41251
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:32877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:01.462453 13581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:01.462798 13581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:01.462962 13581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:01.470307 13581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:01.470472 13581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:01.470669 13581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:01.480827 13581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:32877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:01.482216 13581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:01.483601 13581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:01.497639 13587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:01.497660 13586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:01.498356 13589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:01.498584 13581 server_base.cc:1034] running on GCE node
I20260928 15:44:01.499212 13581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:01.500258 13581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:01.501482 13581 hybrid_clock.cc:648] HybridClock initialized: now 1790610241501407 us; error 63 us; skew 500 ppm
I20260928 15:44:01.501861 13581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:01 dist-test-slave-j968 krb5kdc[13340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610241, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:01.509208 13581 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:01.510895 13581 webserver.cc:460] Webserver started at http://127.13.5.66:39265/ using document root <none> and password file <none>
I20260928 15:44:01.511685 13581 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:01.511837 13581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:01.512198 13581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:01.515400 13581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "35f811ff68044f12b96ca59a887dcb08"
format_stamp: "Formatted at 2026-09-28 15:44:01 on dist-test-slave-j968"
server_key: "451d0118d0c5759678c342704eaa1a5c"
server_key_iv: "6a2eeeb08db968cdf42d0ef8d4c04fb7"
server_key_version: "encryptionkey@0"
I20260928 15:44:01.516793 13581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "35f811ff68044f12b96ca59a887dcb08"
format_stamp: "Formatted at 2026-09-28 15:44:01 on dist-test-slave-j968"
server_key: "451d0118d0c5759678c342704eaa1a5c"
server_key_iv: "6a2eeeb08db968cdf42d0ef8d4c04fb7"
server_key_version: "encryptionkey@0"
I20260928 15:44:01.523180 13581 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:44:01.527702 13596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:01.529428 13581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:01.529633 13581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "35f811ff68044f12b96ca59a887dcb08"
format_stamp: "Formatted at 2026-09-28 15:44:01 on dist-test-slave-j968"
server_key: "451d0118d0c5759678c342704eaa1a5c"
server_key_iv: "6a2eeeb08db968cdf42d0ef8d4c04fb7"
server_key_version: "encryptionkey@0"
I20260928 15:44:01.529856 13581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:01.550089 13581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:01.555444 13581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:01.555756 13581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:01.556875 13581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:01.558243 13581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:01.558346 13581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:01.558502 13581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:01.558576 13581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:01.593982 13581 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:41729
I20260928 15:44:01.594080 13708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:41729 every 8 connection(s)
I20260928 15:44:01.595353 13581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:01.597927 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 13581
I20260928 15:44:01.598137 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:01.598712 13333 external_mini_cluster.cc:1428] Setting key 6f372b32faef5fbc52e9685a64803076
Sep 28 15:44:01 dist-test-slave-j968 krb5kdc[13340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610241, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:44:01.618181 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:01.598513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51891 (local address 127.13.5.126:41251)
0928 15:44:01.598694 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:01.598699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:01.600351 (+ 1652us) server_negotiation.cc:407] Connection header received
0928 15:44:01.602378 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:01.602384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:01.602479 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:01.602623 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:01.604486 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:01.605213 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:01.606937 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:01.607387 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:01.607531 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:01.611640 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:01.611646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:01.611651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:01.611688 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:01.614579 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:01.615612 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:01.615617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:01.615619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:01.615755 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:01.616319 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:01.616323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:01.616325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:01.616648 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:01.616815 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:01.617585 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 15:44:01.617838 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
I20260928 15:44:01.619242 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:01.599202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:41251 (local address 127.13.5.66:51891)
0928 15:44:01.600173 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:01.600228 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:01.601968 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:01.602867 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:01.602876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:01.603560 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:01.604298 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:01.604318 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:01.605394 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:01.605399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:01.606781 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:01.606794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:01.607634 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:01.607641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:01.607949 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:01.608678 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:01.608706 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:01.611290 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:01.614747 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:01.614757 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:01.614778 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:01.615477 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:01.615916 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:01.615923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:01.615926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:01.616140 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:01.616781 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:01.616789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:01.617191 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 15:44:01.617471 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 15:44:01.618058 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":398,"threads_started":1}
I20260928 15:44:01.620805 13709 heartbeater.cc:347] Connected to a master server at 127.13.5.126:41251
I20260928 15:44:01.621171 13709 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:01.622116 13709 heartbeater.cc:511] Master 127.13.5.126:41251 requested a full tablet report, sending...
I20260928 15:44:01.623843 13385 ts_manager.cc:194] Registered new tserver with Master: 35f811ff68044f12b96ca59a887dcb08 (127.13.5.66:41729)
I20260928 15:44:01.625228 13385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:51891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:01.643462 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41251
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:32877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:01.996178 13714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:01.996513 13714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:01.996611 13714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:02.004436 13714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:02.004551 13714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:02.004668 13714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:02.014629 13714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:32877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41251
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:02.015959 13714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:02.017277 13714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:02.030766 13720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:02.030776 13719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:02.031037 13722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:02.031606 13714 server_base.cc:1034] running on GCE node
I20260928 15:44:02.032276 13714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:02.033234 13714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:02.034489 13714 hybrid_clock.cc:648] HybridClock initialized: now 1790610242034408 us; error 54 us; skew 500 ppm
I20260928 15:44:02.034875 13714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610242, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:02.038092 13714 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:02.039595 13714 webserver.cc:460] Webserver started at http://127.13.5.67:42059/ using document root <none> and password file <none>
I20260928 15:44:02.040369 13714 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:02.040517 13714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:02.040840 13714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:02.043813 13714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "d397f8c97b5541a9b8b7f0d3eb56d09b"
format_stamp: "Formatted at 2026-09-28 15:44:02 on dist-test-slave-j968"
server_key: "87529106fa8995e570e0336ad3d52eb3"
server_key_iv: "7c365ba093f454f51c14eed5dbd03f2f"
server_key_version: "encryptionkey@0"
I20260928 15:44:02.044683 13714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "d397f8c97b5541a9b8b7f0d3eb56d09b"
format_stamp: "Formatted at 2026-09-28 15:44:02 on dist-test-slave-j968"
server_key: "87529106fa8995e570e0336ad3d52eb3"
server_key_iv: "7c365ba093f454f51c14eed5dbd03f2f"
server_key_version: "encryptionkey@0"
I20260928 15:44:02.050149 13714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:44:02.053714 13729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:02.055040 13714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:44:02.055228 13714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "d397f8c97b5541a9b8b7f0d3eb56d09b"
format_stamp: "Formatted at 2026-09-28 15:44:02 on dist-test-slave-j968"
server_key: "87529106fa8995e570e0336ad3d52eb3"
server_key_iv: "7c365ba093f454f51c14eed5dbd03f2f"
server_key_version: "encryptionkey@0"
I20260928 15:44:02.055434 13714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:02.072216 13714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:02.076839 13714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:02.077217 13714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:02.078275 13714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:02.079641 13714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:02.079741 13714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:02.079941 13714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:02.080024 13714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 15:44:02.111171 13575 heartbeater.cc:503] Master 127.13.5.126:41251 was elected leader, sending a full tablet report...
I20260928 15:44:02.114791 13714 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:43103
I20260928 15:44:02.114876 13841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:43103 every 8 connection(s)
I20260928 15:44:02.116206 13714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:02.121376 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 13714
I20260928 15:44:02.121613 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:02.122038 13333 external_mini_cluster.cc:1428] Setting key ad78bb2cd0a3bfcf5aca1940f9ff0499
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610242, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:02.140722 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:02.118900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:50385 (local address 127.13.5.126:41251)
0928 15:44:02.119440 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:02.119445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:02.120415 (+ 970us) server_negotiation.cc:407] Connection header received
0928 15:44:02.122501 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:02.122506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:02.122599 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:02.122722 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:02.124634 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.125706 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.127635 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.128186 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.128307 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:02.132058 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:02.132064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:02.132067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:02.132105 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:02.136454 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.137639 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.137647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.137654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.137855 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.138523 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.138529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.138532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.139103 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:02.139277 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:02.140089 (+ 812us) server_negotiation.cc:299] Negotiation successful
0928 15:44:02.140310 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260928 15:44:02.141952 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:02.119408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:41251 (local address 127.13.5.67:50385)
0928 15:44:02.120248 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:02.120293 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:02.122195 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:02.123013 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:02.123026 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:02.123707 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:02.124453 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:02.124481 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.125907 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.125912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:02.127470 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:02.127484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.128411 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.128416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:02.128632 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:02.129351 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:02.129374 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:02.131745 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:02.136669 (+ 4924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:02.136679 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:02.136701 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:02.137442 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:02.138003 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:02.138009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:02.138013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:02.138346 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:02.139252 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:02.139260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:02.139666 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 15:44:02.140001 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 15:44:02.140553 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":301,"threads_started":1}
I20260928 15:44:02.143426 13842 heartbeater.cc:347] Connected to a master server at 127.13.5.126:41251
I20260928 15:44:02.143715 13842 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:02.144503 13842 heartbeater.cc:511] Master 127.13.5.126:41251 requested a full tablet report, sending...
I20260928 15:44:02.145936 13385 ts_manager.cc:194] Registered new tserver with Master: d397f8c97b5541a9b8b7f0d3eb56d09b (127.13.5.67:43103)
I20260928 15:44:02.146767 13385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:50385
I20260928 15:44:02.151367 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:02.196949 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:02.181248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.13.5.65:42369)
0928 15:44:02.181950 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:02.181957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:02.181985 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:44:02.182114 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:02.182121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:02.182345 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:02.182475 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:02.183991 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.184993 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.186264 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.186826 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.186923 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:02.190952 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:02.190964 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:02.190979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:02.191034 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:02.194117 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.194758 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.194766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.194768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.194897 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.195272 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.195276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.195278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.195689 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:02.196019 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:02.196313 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 15:44:02.196623 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":397,"threads_started":1}
W20260928 15:44:02.198460 13490 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48636
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:02.243806 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:02.229736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.13.5.65:42369)
0928 15:44:02.229951 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:02.229956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:02.230044 (+ 88us) server_negotiation.cc:407] Connection header received
0928 15:44:02.230194 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:02.230198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:02.230264 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:02.230353 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:02.232047 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.232739 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.234093 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.234570 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.234665 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:02.238794 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:02.238799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:02.238802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:02.238841 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:02.241561 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.242186 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.242189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.242192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.242308 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.242625 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.242628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.242630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.242869 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:02.243036 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:02.243295 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:44:02.243458 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:44:02.245529 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 13447
I20260928 15:44:02.258430 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:02.246549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.13.5.65:42369)
0928 15:44:02.246771 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:02.246779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:02.247068 (+ 289us) server_negotiation.cc:407] Connection header received
0928 15:44:02.247250 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:02.247255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:02.247317 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:02.247414 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:02.248644 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.249359 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.250583 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.251008 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.251144 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:02.253420 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:02.253425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:02.253428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:02.253459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:02.255940 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.256611 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.256617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.256623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.256798 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.257177 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.257181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.257183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.257445 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:02.257644 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:02.257891 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 15:44:02.258058 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:44:02.259161 13570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:02.344568 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 13447
I20260928 15:44:02.362105 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 13581
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:44:02.378965 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:02.363093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.13.5.66:41729)
0928 15:44:02.363633 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:02.363640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:02.363667 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:44:02.363729 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:02.363735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:02.364033 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:02.364177 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:02.365606 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.366634 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.367965 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.368591 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.368706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:02.372802 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:02.372817 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:02.372832 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:02.372891 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:02.376027 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.376813 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.376822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.376824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.376992 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.377353 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.377356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.377359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.377767 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:02.378077 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:02.378350 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 15:44:02.378581 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260928 15:44:02.380477 13704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:02.463912 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 13581
I20260928 15:44:02.480818 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 13714
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:44:02.497241 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:02.481806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.13.5.67:43103)
0928 15:44:02.482481 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:02.482487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:02.482518 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:44:02.482580 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:02.482588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:02.482805 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:02.482938 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:02.484231 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.485285 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.486597 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:02.487193 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:02.487298 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:02.491424 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:02.491444 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:02.491459 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:02.491494 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:02.494583 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.495204 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.495210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.495213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.495326 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:02.495666 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:02.495669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:02.495671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:02.496075 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:02.496316 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:02.496557 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 15:44:02.496760 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":348,"threads_started":1}
I20260928 15:44:02.498718 13837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:02.583841 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 13714
I20260928 15:44:02.600575 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 13353
I20260928 15:44:02.602116 13415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:02.706804 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 13353
2026-09-28T15:44:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3994 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13864](info): set up 2 sockets
Sep 28 15:44:02 dist-test-slave-j968 krb5kdc[13864](info): commencing operation
krb5kdc: starting...
W20260928 15:44:03.810755 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:03 dist-test-slave-j968 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:04.005300 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:34779
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:04.337688 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:04.338001 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:04.338162 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:04.345624 13877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:04.345750 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:04.345839 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:04.345938 13877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:04.346043 13877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:04.356585 13877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:34779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:04.358022 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:04.359401 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:04.371786 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:04.371707 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:04.371814 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:04.372962 13877 server_base.cc:1034] running on GCE node
I20260928 15:44:04.373818 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:04.375402 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:04.376664 13877 hybrid_clock.cc:648] HybridClock initialized: now 1790610244376607 us; error 46 us; skew 500 ppm
I20260928 15:44:04.377111 13877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:04 dist-test-slave-j968 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610244, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:04.387048 13877 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:04.388783 13877 webserver.cc:460] Webserver started at http://127.13.5.126:46775/ using document root <none> and password file <none>
I20260928 15:44:04.389580 13877 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:04.389691 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:04.389967 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:04.393289 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "fda33b32fbec42daa796d36bcede1b5e"
format_stamp: "Formatted at 2026-09-28 15:44:04 on dist-test-slave-j968"
server_key: "baca7e293279669e186bfc7c4a4ed8ea"
server_key_iv: "267020945bc54a65040b29869284e420"
server_key_version: "encryptionkey@0"
I20260928 15:44:04.394062 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "fda33b32fbec42daa796d36bcede1b5e"
format_stamp: "Formatted at 2026-09-28 15:44:04 on dist-test-slave-j968"
server_key: "baca7e293279669e186bfc7c4a4ed8ea"
server_key_iv: "267020945bc54a65040b29869284e420"
server_key_version: "encryptionkey@0"
I20260928 15:44:04.399679 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:44:04.403750 13892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:04.405376 13877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:04.405544 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "fda33b32fbec42daa796d36bcede1b5e"
format_stamp: "Formatted at 2026-09-28 15:44:04 on dist-test-slave-j968"
server_key: "baca7e293279669e186bfc7c4a4ed8ea"
server_key_iv: "267020945bc54a65040b29869284e420"
server_key_version: "encryptionkey@0"
I20260928 15:44:04.405722 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:04.439939 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:04.444649 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:04.444908 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:04.464351 13877 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:34779
I20260928 15:44:04.464382 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:34779 every 8 connection(s)
I20260928 15:44:04.465893 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:04.469879 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:04.474450 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 13877
I20260928 15:44:04.474697 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:04.475545 13333 external_mini_cluster.cc:1428] Setting key 90e0540318534cb43241d6566064f2c0
I20260928 15:44:04.482013 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e: Bootstrap starting.
I20260928 15:44:04.485775 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:04.486949 13944 log.cc:824] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:04.490022 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e: No bootstrap required, opened a new log
Sep 28 15:44:04 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:04.498126 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda33b32fbec42daa796d36bcede1b5e" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34779 } }
I20260928 15:44:04.498600 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:04.498729 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda33b32fbec42daa796d36bcede1b5e, State: Initialized, Role: FOLLOWER
I20260928 15:44:04.499352 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda33b32fbec42daa796d36bcede1b5e" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34779 } }
I20260928 15:44:04.499619 13944 raft_consensus.cc:388] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:04.499785 13944 raft_consensus.cc:482] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:04.499841 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:04.477741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.13.5.126:34779)
0928 15:44:04.478594 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:04.478610 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:04.478658 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:44:04.480531 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:04.480554 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:04.480951 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:04.481282 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:04.483158 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:04.484569 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:04.486320 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:04.486989 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:04.487198 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:04.491367 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:04.491376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:04.491391 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:04.491426 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:04.494669 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:04.495373 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:04.495380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:04.495386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:04.495564 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:04.495986 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:04.496009 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:04.496011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:04.496606 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:04.497201 (+ 595us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:04.497782 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 15:44:04.498286 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":391,"threads_started":1}
I20260928 15:44:04.500010 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:04.502728 13944 raft_consensus.cc:504] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda33b32fbec42daa796d36bcede1b5e" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34779 } }
I20260928 15:44:04.503331 13944 leader_election.cc:302] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fda33b32fbec42daa796d36bcede1b5e; no voters:
I20260928 15:44:04.503988 13944 leader_election.cc:288] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:04.504129 13949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:04.504865 13949 raft_consensus.cc:686] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [term 1 LEADER]: Becoming Leader. State: Replica: fda33b32fbec42daa796d36bcede1b5e, State: Running, Role: LEADER
I20260928 15:44:04.505359 13949 consensus_queue.cc:237] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda33b32fbec42daa796d36bcede1b5e" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34779 } }
I20260928 15:44:04.506102 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:04.508422 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fda33b32fbec42daa796d36bcede1b5e. Latest consensus state: current_term: 1 leader_uuid: "fda33b32fbec42daa796d36bcede1b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda33b32fbec42daa796d36bcede1b5e" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34779 } } }
I20260928 15:44:04.508800 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:04.509481 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fda33b32fbec42daa796d36bcede1b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fda33b32fbec42daa796d36bcede1b5e" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34779 } } }
I20260928 15:44:04.509905 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:04.511451 13957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:04.518007 13957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:04.527824 13957 catalog_manager.cc:1347] Generated new cluster ID: ff5cd8eb7cb14ee2a8945a3dfe8b62a7
I20260928 15:44:04.527993 13957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:04.550386 13957 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:04.552386 13957 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:04.568516 13957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fda33b32fbec42daa796d36bcede1b5e: Generated new TSK 0
I20260928 15:44:04.569661 13957 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:04.625552 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:34779
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:04.968218 13970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:04.968591 13970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:04.968786 13970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:04.976303 13970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:04.976414 13970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:04.976627 13970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:04.986645 13970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:34779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:04.988052 13970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:04.989639 13970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:05.003842 13975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:05.003923 13978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:05.003774 13976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:05.009986 13970 server_base.cc:1034] running on GCE node
I20260928 15:44:05.010793 13970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:05.011998 13970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:05.013204 13970 hybrid_clock.cc:648] HybridClock initialized: now 1790610245013167 us; error 38 us; skew 500 ppm
I20260928 15:44:05.013624 13970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:05 dist-test-slave-j968 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:05.017256 13970 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:05.019093 13970 webserver.cc:460] Webserver started at http://127.13.5.65:43547/ using document root <none> and password file <none>
I20260928 15:44:05.019968 13970 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:05.020105 13970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:05.020462 13970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:05.023732 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "95d549ab49714e398954a518f63e6180"
format_stamp: "Formatted at 2026-09-28 15:44:05 on dist-test-slave-j968"
server_key: "e35a3e9a697ec0cce41602acab3136f0"
server_key_iv: "be3ee935529838db50134ce32cc028b2"
server_key_version: "encryptionkey@0"
I20260928 15:44:05.024645 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "95d549ab49714e398954a518f63e6180"
format_stamp: "Formatted at 2026-09-28 15:44:05 on dist-test-slave-j968"
server_key: "e35a3e9a697ec0cce41602acab3136f0"
server_key_iv: "be3ee935529838db50134ce32cc028b2"
server_key_version: "encryptionkey@0"
I20260928 15:44:05.030216 13970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 15:44:05.033954 13985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:05.035324 13970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:44:05.035490 13970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "95d549ab49714e398954a518f63e6180"
format_stamp: "Formatted at 2026-09-28 15:44:05 on dist-test-slave-j968"
server_key: "e35a3e9a697ec0cce41602acab3136f0"
server_key_iv: "be3ee935529838db50134ce32cc028b2"
server_key_version: "encryptionkey@0"
I20260928 15:44:05.035701 13970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:05.055531 13970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:05.060328 13970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:05.060657 13970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:05.061761 13970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:05.063405 13970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:05.063480 13970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:05.063632 13970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:05.063689 13970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:05.099771 13970 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:46077
I20260928 15:44:05.099826 14097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:46077 every 8 connection(s)
I20260928 15:44:05.101184 13970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:05.105489 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 13970
I20260928 15:44:05.105736 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:05.106595 13333 external_mini_cluster.cc:1428] Setting key c97014b04354eae6ce3c2886811b1cda
Sep 28 15:44:05 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:44:05.126892 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:05.104038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:37027 (local address 127.13.5.126:34779)
0928 15:44:05.104530 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:05.104535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:05.105837 (+ 1302us) server_negotiation.cc:407] Connection header received
0928 15:44:05.108271 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:05.108277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:05.108352 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:05.108468 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:05.110764 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:05.111501 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:05.114188 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:05.114962 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:05.115126 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:05.119060 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:05.119070 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:05.119079 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:05.119131 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:05.123121 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:05.124295 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:05.124303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:05.124310 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:05.124483 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:05.125015 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:05.125020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:05.125024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:05.125415 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:05.125595 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:05.126393 (+ 798us) server_negotiation.cc:299] Negotiation successful
0928 15:44:05.126632 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":260,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
I20260928 15:44:05.128482 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:05.104541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:34779 (local address 127.13.5.65:37027)
0928 15:44:05.105581 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:05.105638 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:05.107981 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:05.108821 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:05.108830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:05.109690 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:05.110577 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:05.110608 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:05.111690 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:05.111699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:05.114041 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:05.114055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:05.115234 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:05.115240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:05.115540 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:05.116307 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:05.116332 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:05.118778 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:05.123331 (+ 4553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:05.123339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:05.123355 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:05.124092 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:05.124641 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:05.124650 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:05.124653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:05.124871 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:05.125577 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:05.125584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:05.125959 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 15:44:05.126281 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 15:44:05.126784 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":391,"threads_started":1}
I20260928 15:44:05.130266 14098 heartbeater.cc:347] Connected to a master server at 127.13.5.126:34779
I20260928 15:44:05.130600 14098 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:05.131377 14098 heartbeater.cc:511] Master 127.13.5.126:34779 requested a full tablet report, sending...
I20260928 15:44:05.133845 13909 ts_manager.cc:194] Registered new tserver with Master: 95d549ab49714e398954a518f63e6180 (127.13.5.65:46077)
I20260928 15:44:05.136139 13909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:37027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:05.152364 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:34779
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:05.489364 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:05.489643 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:05.489739 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:05.496997 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:05.497107 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:05.497220 14104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:05.507088 14104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:34779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:05.508508 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:05.510013 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:05.523944 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:05.524044 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:05.524175 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:05.524822 14104 server_base.cc:1034] running on GCE node
I20260928 15:44:05.525472 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:05.526530 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:05.527771 14104 hybrid_clock.cc:648] HybridClock initialized: now 1790610245527710 us; error 51 us; skew 500 ppm
I20260928 15:44:05.528131 14104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:05 dist-test-slave-j968 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:05.531371 14104 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:05.532936 14104 webserver.cc:460] Webserver started at http://127.13.5.66:35801/ using document root <none> and password file <none>
I20260928 15:44:05.533659 14104 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:05.533802 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:05.534116 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:05.537103 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "a547beca7c144f539b3bc798373ca1e9"
format_stamp: "Formatted at 2026-09-28 15:44:05 on dist-test-slave-j968"
server_key: "1e0edc386064318b58d09c17a1229012"
server_key_iv: "0d92ed924ec8ef36618890d10283dada"
server_key_version: "encryptionkey@0"
I20260928 15:44:05.537884 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "a547beca7c144f539b3bc798373ca1e9"
format_stamp: "Formatted at 2026-09-28 15:44:05 on dist-test-slave-j968"
server_key: "1e0edc386064318b58d09c17a1229012"
server_key_iv: "0d92ed924ec8ef36618890d10283dada"
server_key_version: "encryptionkey@0"
I20260928 15:44:05.543242 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:44:05.546869 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:05.548401 14104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:05.548581 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "a547beca7c144f539b3bc798373ca1e9"
format_stamp: "Formatted at 2026-09-28 15:44:05 on dist-test-slave-j968"
server_key: "1e0edc386064318b58d09c17a1229012"
server_key_iv: "0d92ed924ec8ef36618890d10283dada"
server_key_version: "encryptionkey@0"
I20260928 15:44:05.548790 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:05.583554 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:05.588117 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:05.588425 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:05.589458 14104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:05.590991 14104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:05.591077 14104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:05.591166 14104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:05.591209 14104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:05.626221 14104 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:34285
I20260928 15:44:05.626267 14231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:34285 every 8 connection(s)
I20260928 15:44:05.627590 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:05.631058 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 14104
I20260928 15:44:05.631276 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:05.631883 13333 external_mini_cluster.cc:1428] Setting key 3424f6124a4e1ba172fab63d8b08ba38
Sep 28 15:44:05 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610245, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:44:05.649030 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:05.630342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:41527 (local address 127.13.5.126:34779)
0928 15:44:05.630854 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:05.630859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:05.632004 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 15:44:05.634173 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:05.634179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:05.634269 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:05.634384 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:05.635962 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:05.636660 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:05.638194 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:05.638669 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:05.638797 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:05.642929 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:05.642935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:05.642938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:05.642971 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:05.645719 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:05.646474 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:05.646483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:05.646488 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:05.646628 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:05.647084 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:05.647089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:05.647092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:05.647420 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:05.647572 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:05.648410 (+ 838us) server_negotiation.cc:299] Negotiation successful
0928 15:44:05.648636 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":295,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
I20260928 15:44:05.650169 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:05.630871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:34779 (local address 127.13.5.66:41527)
0928 15:44:05.631783 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:05.631828 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:05.633873 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:05.634646 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:05.634654 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:05.635170 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:05.635754 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:05.635771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:05.636818 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:05.636823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:05.638050 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:05.638062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:05.638893 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:05.638901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:05.639132 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:05.639920 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:05.639960 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:05.642687 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:05.645878 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:05.645887 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:05.645905 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:05.646358 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:05.646748 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:05.646753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:05.646756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:05.646964 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:05.647546 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:05.647554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:05.647977 (+ 423us) client_negotiation.cc:769] Sending connection context
0928 15:44:05.648260 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 15:44:05.648820 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":468,"threads_started":1}
I20260928 15:44:05.651645 14232 heartbeater.cc:347] Connected to a master server at 127.13.5.126:34779
I20260928 15:44:05.652029 14232 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:05.652998 14232 heartbeater.cc:511] Master 127.13.5.126:34779 requested a full tablet report, sending...
I20260928 15:44:05.654768 13909 ts_manager.cc:194] Registered new tserver with Master: a547beca7c144f539b3bc798373ca1e9 (127.13.5.66:34285)
I20260928 15:44:05.655628 13909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:41527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:05.676273 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:34779
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:06.004573 14238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:06.004856 14238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:06.005048 14238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:06.012605 14238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:06.012715 14238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:06.012946 14238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:06.023098 14238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:34779
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:06.024426 14238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:06.025866 14238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:06.039604 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:06.039531 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:06.040102 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:06.040599 14238 server_base.cc:1034] running on GCE node
I20260928 15:44:06.041229 14238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:06.042187 14238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:06.043416 14238 hybrid_clock.cc:648] HybridClock initialized: now 1790610246043348 us; error 57 us; skew 500 ppm
I20260928 15:44:06.043726 14238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610246, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:06.047012 14238 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:06.048652 14238 webserver.cc:460] Webserver started at http://127.13.5.67:44525/ using document root <none> and password file <none>
I20260928 15:44:06.049360 14238 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:06.049489 14238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:06.049832 14238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:06.052886 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "8b7f2362fb27425cb86325418132ef03"
format_stamp: "Formatted at 2026-09-28 15:44:06 on dist-test-slave-j968"
server_key: "9480172c61a036dbb76f18901e5e9519"
server_key_iv: "08cdf5479a5b0dd02d10c23bcada1f38"
server_key_version: "encryptionkey@0"
I20260928 15:44:06.053725 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "8b7f2362fb27425cb86325418132ef03"
format_stamp: "Formatted at 2026-09-28 15:44:06 on dist-test-slave-j968"
server_key: "9480172c61a036dbb76f18901e5e9519"
server_key_iv: "08cdf5479a5b0dd02d10c23bcada1f38"
server_key_version: "encryptionkey@0"
I20260928 15:44:06.058943 14238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:44:06.062700 14253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:06.064277 14238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:06.064416 14238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "8b7f2362fb27425cb86325418132ef03"
format_stamp: "Formatted at 2026-09-28 15:44:06 on dist-test-slave-j968"
server_key: "9480172c61a036dbb76f18901e5e9519"
server_key_iv: "08cdf5479a5b0dd02d10c23bcada1f38"
server_key_version: "encryptionkey@0"
I20260928 15:44:06.064592 14238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:06.090070 14238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:06.094394 14238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:06.094648 14238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:06.095659 14238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:06.096987 14238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:06.097059 14238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:06.097139 14238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:06.097183 14238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:06.132112 14238 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:35647
I20260928 15:44:06.132225 14365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:35647 every 8 connection(s)
I20260928 15:44:06.133987 14238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:06.134513 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 14238
I20260928 15:44:06.134734 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:06.135375 13333 external_mini_cluster.cc:1428] Setting key beaa3d064b8a1cf19d4532ba3474bf33
I20260928 15:44:06.139508 14098 heartbeater.cc:503] Master 127.13.5.126:34779 was elected leader, sending a full tablet report...
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610246, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:06.156844 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.137734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:35177 (local address 127.13.5.126:34779)
0928 15:44:06.137970 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.137976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.139550 (+ 1574us) server_negotiation.cc:407] Connection header received
0928 15:44:06.141876 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.141881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.141985 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.142082 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:06.144357 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.145203 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.146791 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.147203 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.147298 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.150952 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:06.150957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:06.150965 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:06.150997 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:06.154020 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.154728 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.154731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.154734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.154848 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.155238 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.155241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.155243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.155518 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:06.155636 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.156406 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.156606 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:44:06.157938 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.138382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:34779 (local address 127.13.5.67:35177)
0928 15:44:06.139392 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:06.139444 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:06.141609 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:06.142372 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:06.142382 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:06.143286 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:06.144166 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.144190 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.145347 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.145351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.146653 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.146667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.147398 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.147403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.147612 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.148377 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:06.148403 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:06.150737 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:06.154150 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:06.154157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:06.154172 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:06.154608 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:06.154937 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:06.154941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:06.154943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:06.155139 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:06.155633 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:06.155639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:06.156058 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 15:44:06.156299 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 15:44:06.156723 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":403,"threads_started":1}
I20260928 15:44:06.159353 14366 heartbeater.cc:347] Connected to a master server at 127.13.5.126:34779
I20260928 15:44:06.159703 14366 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:06.160519 14366 heartbeater.cc:511] Master 127.13.5.126:34779 requested a full tablet report, sending...
I20260928 15:44:06.162009 13909 ts_manager.cc:194] Registered new tserver with Master: 8b7f2362fb27425cb86325418132ef03 (127.13.5.67:35647)
I20260928 15:44:06.162817 13909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:35177
I20260928 15:44:06.165450 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:44:06.185876 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.172354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.13.5.126:34779)
0928 15:44:06.172638 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.172648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.173093 (+ 445us) server_negotiation.cc:407] Connection header received
0928 15:44:06.173312 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.173318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.173411 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.173521 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:06.175069 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.175825 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.177182 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.177623 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.177764 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.180184 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:06.180190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:06.180194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:06.180236 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:06.183003 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.183694 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.183699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.183703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.183841 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.184280 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.184285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.184289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.184655 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:06.184928 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.185290 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.185527 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 15:44:06.192831 13909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:06.199947 13909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:06.224038 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.214100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.13.5.67:35647)
0928 15:44:06.215786 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.215797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.215832 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:44:06.215957 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.215968 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.216239 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.216435 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:06.217058 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.218508 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.221469 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.222748 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.222910 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.223177 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.223451 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.223624 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1461,"thread_start_us":1248,"threads_started":1}
I20260928 15:44:06.224107 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.213679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:35647 (local address 127.0.0.1:44144)
0928 15:44:06.214909 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:06.214953 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:06.215114 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:06.216443 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:06.216450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:06.216480 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:06.216883 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.216897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.218645 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.218652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.221262 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.221278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.222958 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.222964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.223202 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.223259 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:44:06.223407 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 15:44:06.223613 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":1,"thread_start_us":791,"threads_started":1}
I20260928 15:44:06.228401 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.215536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:34285 (local address 127.0.0.1:56792)
0928 15:44:06.216097 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:06.216116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:06.216221 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:06.219402 (+ 3181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:06.219410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:06.219445 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:06.220045 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.220065 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.222016 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.222019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.225284 (+ 3265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.225303 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.227481 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.227488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.227740 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.227774 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:44:06.227861 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:44:06.228023 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":23,"thread_start_us":271,"threads_started":1}
I20260928 15:44:06.228518 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.215914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.13.5.66:34285)
0928 15:44:06.218783 (+ 2869us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.218793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.218822 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:44:06.218929 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.218937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.219185 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.219399 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:06.220329 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.221823 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.225472 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.226990 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.227094 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.227295 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.227988 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.228160 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":891,"threads_started":1}
I20260928 15:44:06.228196 14301 tablet_service.cc:1467] Processing CreateTablet for tablet a43d44d94b734fdd88f680a90c5821e6 (DEFAULT_TABLE table=test-table [id=7855e3985a4e408f94f782a650d8755a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:06.229620 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.216156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:46077 (local address 127.0.0.1:52506)
0928 15:44:06.220510 (+ 4354us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:06.220527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:06.220677 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:06.221633 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:06.221638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:06.221665 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:06.222018 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.222030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.223982 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.223987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.226144 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.226158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.228812 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.228818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.229010 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.229050 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:44:06.229150 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:44:06.229282 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4234,"thread_start_us":4086,"threads_started":1}
I20260928 15:44:06.229879 14301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43d44d94b734fdd88f680a90c5821e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:06.229928 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.216852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.13.5.65:46077)
0928 15:44:06.220090 (+ 3238us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.220099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.220802 (+ 703us) server_negotiation.cc:407] Connection header received
0928 15:44:06.220903 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.220910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.221138 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.221319 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:06.222195 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.223791 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.226311 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.227766 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.227925 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.228155 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.229211 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.229385 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3068,"thread_start_us":2903,"threads_started":1}
I20260928 15:44:06.232303 14167 tablet_service.cc:1467] Processing CreateTablet for tablet a43d44d94b734fdd88f680a90c5821e6 (DEFAULT_TABLE table=test-table [id=7855e3985a4e408f94f782a650d8755a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:06.233822 14167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43d44d94b734fdd88f680a90c5821e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:06.235077 14033 tablet_service.cc:1467] Processing CreateTablet for tablet a43d44d94b734fdd88f680a90c5821e6 (DEFAULT_TABLE table=test-table [id=7855e3985a4e408f94f782a650d8755a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:06.237053 14033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43d44d94b734fdd88f680a90c5821e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:06.245666 14384 tablet_bootstrap.cc:492] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03: Bootstrap starting.
I20260928 15:44:06.248560 14385 tablet_bootstrap.cc:492] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9: Bootstrap starting.
I20260928 15:44:06.249610 14384 tablet_bootstrap.cc:654] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:06.250219 14386 tablet_bootstrap.cc:492] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180: Bootstrap starting.
I20260928 15:44:06.251087 14384 log.cc:824] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:06.252267 14385 tablet_bootstrap.cc:654] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:06.253532 14385 log.cc:824] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:06.254048 14386 tablet_bootstrap.cc:654] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:06.255563 14386 log.cc:824] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:06.260134 14384 tablet_bootstrap.cc:492] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03: No bootstrap required, opened a new log
I20260928 15:44:06.260282 14385 tablet_bootstrap.cc:492] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9: No bootstrap required, opened a new log
I20260928 15:44:06.260283 14386 tablet_bootstrap.cc:492] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180: No bootstrap required, opened a new log
I20260928 15:44:06.260520 14384 ts_tablet_manager.cc:1397] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260928 15:44:06.260565 14386 ts_tablet_manager.cc:1397] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260928 15:44:06.260572 14385 ts_tablet_manager.cc:1397] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260928 15:44:06.269007 14384 raft_consensus.cc:348] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.269507 14384 raft_consensus.cc:374] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:06.269606 14384 raft_consensus.cc:729] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7f2362fb27425cb86325418132ef03, State: Initialized, Role: FOLLOWER
I20260928 15:44:06.270268 14384 consensus_queue.cc:260] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.271597 14385 raft_consensus.cc:348] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.271595 14386 raft_consensus.cc:348] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.272099 14385 raft_consensus.cc:374] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:06.272097 14386 raft_consensus.cc:374] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:06.272205 14385 raft_consensus.cc:729] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a547beca7c144f539b3bc798373ca1e9, State: Initialized, Role: FOLLOWER
I20260928 15:44:06.272205 14386 raft_consensus.cc:729] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d549ab49714e398954a518f63e6180, State: Initialized, Role: FOLLOWER
I20260928 15:44:06.272866 14385 consensus_queue.cc:260] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.272866 14386 consensus_queue.cc:260] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.277824 14386 ts_tablet_manager.cc:1428] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260928 15:44:06.278009 14384 ts_tablet_manager.cc:1428] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260928 15:44:06.278833 14366 heartbeater.cc:503] Master 127.13.5.126:34779 was elected leader, sending a full tablet report...
I20260928 15:44:06.280817 14232 heartbeater.cc:503] Master 127.13.5.126:34779 was elected leader, sending a full tablet report...
I20260928 15:44:06.283545 14385 ts_tablet_manager.cc:1428] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9: Time spent starting tablet: real 0.023s user 0.012s sys 0.008s
W20260928 15:44:06.354061 14099 tablet.cc:2369] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:06.379805 14233 tablet.cc:2369] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:06.386439 14367 tablet.cc:2369] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:44:06.454186 14390 raft_consensus.cc:482] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:06.454530 14390 raft_consensus.cc:504] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.456326 14390 leader_election.cc:288] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a547beca7c144f539b3bc798373ca1e9 (127.13.5.66:34285), 95d549ab49714e398954a518f63e6180 (127.13.5.65:46077)
I20260928 15:44:06.462601 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.456975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:34285 (local address 127.13.5.67:53183)
0928 15:44:06.457192 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:06.457209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:06.457365 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:06.457783 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:06.457788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:06.457819 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:06.458063 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.458077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.459098 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.459103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.460643 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.460657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.461846 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.461852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.462039 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.462065 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:44:06.462173 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:44:06.462308 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 15:44:06.462597 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.456814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:53183 (local address 127.13.5.66:34285)
0928 15:44:06.457101 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.457109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.457298 (+ 189us) server_negotiation.cc:407] Connection header received
0928 15:44:06.457528 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.457536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.457629 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.457749 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:06.458231 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.458933 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.460812 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.461674 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.461829 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.461929 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.462202 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.462334 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 15:44:06.463744 14393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.457420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:46077 (local address 127.13.5.67:46975)
0928 15:44:06.458386 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:06.458399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:06.458542 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:06.458860 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:06.458864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:06.458888 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:06.459305 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.459315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.460450 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.460455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.461818 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:06.461830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.463051 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.463060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:06.463233 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.463269 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:44:06.463353 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:44:06.463443 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":622,"threads_started":1}
I20260928 15:44:06.463901 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.457678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:46975 (local address 127.13.5.65:46077)
0928 15:44:06.457902 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.457908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.458499 (+ 591us) server_negotiation.cc:407] Connection header received
0928 15:44:06.458683 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.458687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.458747 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.458843 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:06.459465 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.460281 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.461984 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.462822 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.462957 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.463064 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.463490 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.463635 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 15:44:06.463887 14187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43d44d94b734fdd88f680a90c5821e6" candidate_uuid: "8b7f2362fb27425cb86325418132ef03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a547beca7c144f539b3bc798373ca1e9" is_pre_election: true
I20260928 15:44:06.464602 14187 raft_consensus.cc:2454] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7f2362fb27425cb86325418132ef03 in term 0.
I20260928 15:44:06.464883 14053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43d44d94b734fdd88f680a90c5821e6" candidate_uuid: "8b7f2362fb27425cb86325418132ef03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d549ab49714e398954a518f63e6180" is_pre_election: true
I20260928 15:44:06.465346 14254 leader_election.cc:302] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b7f2362fb27425cb86325418132ef03, a547beca7c144f539b3bc798373ca1e9; no voters:
I20260928 15:44:06.465363 14053 raft_consensus.cc:2454] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7f2362fb27425cb86325418132ef03 in term 0.
I20260928 15:44:06.465747 14390 raft_consensus.cc:2781] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:06.465909 14390 raft_consensus.cc:482] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:06.466027 14390 raft_consensus.cc:3037] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:06.468954 14390 raft_consensus.cc:504] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.469586 14390 leader_election.cc:288] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [CANDIDATE]: Term 1 election: Requested vote from peers a547beca7c144f539b3bc798373ca1e9 (127.13.5.66:34285), 95d549ab49714e398954a518f63e6180 (127.13.5.65:46077)
I20260928 15:44:06.470075 14187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43d44d94b734fdd88f680a90c5821e6" candidate_uuid: "8b7f2362fb27425cb86325418132ef03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a547beca7c144f539b3bc798373ca1e9"
I20260928 15:44:06.470176 14053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43d44d94b734fdd88f680a90c5821e6" candidate_uuid: "8b7f2362fb27425cb86325418132ef03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d549ab49714e398954a518f63e6180"
I20260928 15:44:06.470291 14187 raft_consensus.cc:3037] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:06.470417 14053 raft_consensus.cc:3037] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:06.472672 14187 raft_consensus.cc:2454] T a43d44d94b734fdd88f680a90c5821e6 P a547beca7c144f539b3bc798373ca1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7f2362fb27425cb86325418132ef03 in term 1.
I20260928 15:44:06.473125 14254 leader_election.cc:302] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b7f2362fb27425cb86325418132ef03, a547beca7c144f539b3bc798373ca1e9; no voters:
I20260928 15:44:06.473433 14390 raft_consensus.cc:2781] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:06.473665 14053 raft_consensus.cc:2454] T a43d44d94b734fdd88f680a90c5821e6 P 95d549ab49714e398954a518f63e6180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7f2362fb27425cb86325418132ef03 in term 1.
I20260928 15:44:06.474439 14390 raft_consensus.cc:686] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7f2362fb27425cb86325418132ef03, State: Running, Role: LEADER
I20260928 15:44:06.474931 14390 consensus_queue.cc:237] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } }
I20260928 15:44:06.479708 13908 catalog_manager.cc:5495] T a43d44d94b734fdd88f680a90c5821e6 P 8b7f2362fb27425cb86325418132ef03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7f2362fb27425cb86325418132ef03 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "8b7f2362fb27425cb86325418132ef03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7f2362fb27425cb86325418132ef03" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 35647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a547beca7c144f539b3bc798373ca1e9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95d549ab49714e398954a518f63e6180" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 46077 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:06.528733 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.513533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.13.5.65:46077)
0928 15:44:06.513768 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.513778 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.514198 (+ 420us) server_negotiation.cc:407] Connection header received
0928 15:44:06.514337 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.514341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.514481 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.514570 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:06.516113 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.516923 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.518121 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.518558 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.518715 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.522825 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:06.522839 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:06.522858 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:06.522894 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:06.526069 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.526762 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.526770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.526773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.526915 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.527310 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.527313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.527315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.527686 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:06.527992 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.528290 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.528457 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610246, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610246, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:06.577335 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.562131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.13.5.65:46077)
0928 15:44:06.562340 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.562350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.562722 (+ 372us) server_negotiation.cc:407] Connection header received
0928 15:44:06.562935 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.562940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.563018 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.563128 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:06.564855 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.565609 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.567023 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.567480 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.567629 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.572007 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:06.572012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:06.572016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:06.572058 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:06.574662 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.575321 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.575328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.575332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.575482 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.575932 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.575939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.575946 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.576218 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:06.576478 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.576766 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.576991 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:06.623323 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.608976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.13.5.65:46077)
0928 15:44:06.609285 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.609294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.609600 (+ 306us) server_negotiation.cc:407] Connection header received
0928 15:44:06.609816 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.609821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.609895 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.610007 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:06.611429 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.612213 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.613469 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.613920 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.614056 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.618084 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:06.618089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:06.618092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:06.618127 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:06.620607 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.621373 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.621378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.621380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.621497 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.621866 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.621871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.621874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.622171 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:06.622434 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.622740 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.622980 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 15:44:06.625386 14013 tablet_service.cc:2908] Rejecting scan request for tablet a43d44d94b734fdd88f680a90c5821e6: Uninitialized: clean time has not yet been initialized
I20260928 15:44:06.628914 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 13970
I20260928 15:44:06.642874 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.630405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.13.5.65:46077)
0928 15:44:06.630591 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.630597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.630616 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:44:06.630693 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.630701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.630827 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.630961 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:06.632214 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.633262 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.634415 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.634878 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.635029 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.637350 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:06.637355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:06.637359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:06.637390 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:06.640184 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.640887 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.640899 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.640902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.641024 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.641400 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.641406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.641409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.641711 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:06.642010 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.642310 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.642554 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:44:06.643519 14093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:06.735329 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 13970
I20260928 15:44:06.753546 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14104
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:44:06.769685 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.754402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.13.5.66:34285)
0928 15:44:06.754605 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.754609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.754711 (+ 102us) server_negotiation.cc:407] Connection header received
0928 15:44:06.754866 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.754870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.754936 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.755023 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:06.756345 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.757093 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.758559 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.759074 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.759177 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.763526 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:06.763543 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:06.763561 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:06.763601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:06.766544 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.767332 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.767340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.767342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.767470 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.767938 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.767942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.767944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.768481 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:06.768834 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.769199 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.769409 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:44:06.770428 14227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:06.861425 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 14104
I20260928 15:44:06.879108 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14238
Sep 28 15:44:06 dist-test-slave-j968 krb5kdc[13864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:44:06.894481 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:06.879916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.13.5.67:35647)
0928 15:44:06.880480 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:06.880484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:06.880501 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:44:06.880554 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:06.880558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:06.880617 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:06.880736 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:06.882023 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.882718 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.884038 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:06.884436 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:06.884585 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:06.888754 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:06.888770 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:06.888788 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:06.888839 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:06.891678 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.892389 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.892397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.892400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.892558 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:06.892941 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:06.892947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:06.892949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:06.893311 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:06.893573 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:06.893778 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 15:44:06.893968 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":311,"threads_started":1}
I20260928 15:44:06.895155 14361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:06.988771 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 14238
I20260928 15:44:07.005945 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 13877
I20260928 15:44:07.007330 13939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:07.118093 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 13877
2026-09-28T15:44:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4412 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:07 dist-test-slave-j968 krb5kdc[14414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:07 dist-test-slave-j968 krb5kdc[14414](info): set up 2 sockets
Sep 28 15:44:07 dist-test-slave-j968 krb5kdc[14414](info): commencing operation
krb5kdc: starting...
W20260928 15:44:08.282941 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:08 dist-test-slave-j968 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:08.480136 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43663
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:08.814816 14427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:08.815217 14427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:08.815387 14427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:08.822963 14427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:08.823154 14427 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:44:08.823263 14427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:08.823374 14427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:08.823469 14427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:08.823572 14427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:08.834645 14427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43663
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:08.836059 14427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:08.837462 14427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:08.849992 14435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:08.850015 14432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:08.850050 14433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:08.851096 14427 server_base.cc:1034] running on GCE node
I20260928 15:44:08.851966 14427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:08.853462 14427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:08.854686 14427 hybrid_clock.cc:648] HybridClock initialized: now 1790610248854606 us; error 56 us; skew 500 ppm
I20260928 15:44:08.854987 14427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:08 dist-test-slave-j968 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610248, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:08.858239 14427 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:08.859764 14427 webserver.cc:460] Webserver started at http://127.13.5.126:41957/ using document root <none> and password file <none>
I20260928 15:44:08.860510 14427 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:08.860605 14427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:08.860873 14427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:08.863837 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "5a515e4ed8e7443389fe117a136dda27"
format_stamp: "Formatted at 2026-09-28 15:44:08 on dist-test-slave-j968"
server_key: "46f12be90ac40af541ebd568a3566ea7"
server_key_iv: "84df38aa2c9f8223e38ee0b5d3db363e"
server_key_version: "encryptionkey@0"
I20260928 15:44:08.864629 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "5a515e4ed8e7443389fe117a136dda27"
format_stamp: "Formatted at 2026-09-28 15:44:08 on dist-test-slave-j968"
server_key: "46f12be90ac40af541ebd568a3566ea7"
server_key_iv: "84df38aa2c9f8223e38ee0b5d3db363e"
server_key_version: "encryptionkey@0"
I20260928 15:44:08.869818 14427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:44:08.873636 14442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:08.875236 14427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:08.875496 14427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "5a515e4ed8e7443389fe117a136dda27"
format_stamp: "Formatted at 2026-09-28 15:44:08 on dist-test-slave-j968"
server_key: "46f12be90ac40af541ebd568a3566ea7"
server_key_iv: "84df38aa2c9f8223e38ee0b5d3db363e"
server_key_version: "encryptionkey@0"
I20260928 15:44:08.875711 14427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:08.921401 14427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:08.926105 14427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:08.926344 14427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:08.949002 14427 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43663
I20260928 15:44:08.949054 14503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43663 every 8 connection(s)
I20260928 15:44:08.950423 14427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:08.954313 14504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:08.958300 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 14427
I20260928 15:44:08.958564 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:08.959389 13333 external_mini_cluster.cc:1428] Setting key 6cdb01c320ee20df6bc1ff42897c448d
I20260928 15:44:08.967633 14504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27: Bootstrap starting.
I20260928 15:44:08.970745 14504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:08.971771 14504 log.cc:824] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:44:08 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:08.974748 14504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27: No bootstrap required, opened a new log
I20260928 15:44:08.981989 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:08.961677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.13.5.126:43663)
0928 15:44:08.962495 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:08.962511 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:08.962556 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:44:08.964267 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:08.964289 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:08.964744 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:08.965173 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:08.966603 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:08.968015 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:08.969270 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:08.969927 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:08.970136 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:08.974295 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:08.974304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:08.974321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:08.974356 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:08.977543 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:08.978258 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:08.978265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:08.978272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:08.978429 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:08.978810 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:08.978815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:08.978820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:08.979374 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:08.979688 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:08.980219 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 15:44:08.980691 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":341,"threads_started":1}
I20260928 15:44:08.983322 14504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a515e4ed8e7443389fe117a136dda27" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43663 } }
I20260928 15:44:08.983688 14504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:08.983789 14504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a515e4ed8e7443389fe117a136dda27, State: Initialized, Role: FOLLOWER
I20260928 15:44:08.984412 14504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a515e4ed8e7443389fe117a136dda27" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43663 } }
I20260928 15:44:08.984669 14504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:08.984825 14504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:08.985020 14504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:08.987097 14504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a515e4ed8e7443389fe117a136dda27" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43663 } }
I20260928 15:44:08.987649 14504 leader_election.cc:302] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a515e4ed8e7443389fe117a136dda27; no voters:
I20260928 15:44:08.988400 14504 leader_election.cc:288] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:08.988519 14509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:08.989262 14509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [term 1 LEADER]: Becoming Leader. State: Replica: 5a515e4ed8e7443389fe117a136dda27, State: Running, Role: LEADER
I20260928 15:44:08.989815 14509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a515e4ed8e7443389fe117a136dda27" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43663 } }
I20260928 15:44:08.990708 14504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:08.993680 14511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a515e4ed8e7443389fe117a136dda27. Latest consensus state: current_term: 1 leader_uuid: "5a515e4ed8e7443389fe117a136dda27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a515e4ed8e7443389fe117a136dda27" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43663 } } }
I20260928 15:44:08.994139 14511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:08.995342 14510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a515e4ed8e7443389fe117a136dda27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a515e4ed8e7443389fe117a136dda27" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43663 } } }
I20260928 15:44:08.996153 14510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:08.996801 14518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:09.002034 14518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:09.011699 14518 catalog_manager.cc:1347] Generated new cluster ID: 2bdf0b90536e42dc8b0fff1441b85242
I20260928 15:44:09.011847 14518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:09.023077 14518 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:09.024706 14518 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:09.034370 14518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27: Generated new TSK 0
I20260928 15:44:09.035238 14518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:09.096218 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43663
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:09.442981 14530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:09.443271 14530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:09.443485 14530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:09.451205 14530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:09.451332 14530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:09.451524 14530 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:44:09.451655 14530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:09.462050 14530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:43663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:09.463465 14530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:09.465003 14530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:09.479399 14535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:09.479473 14536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:09.479498 14538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:09.480280 14530 server_base.cc:1034] running on GCE node
I20260928 15:44:09.480988 14530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:09.481985 14530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:09.483234 14530 hybrid_clock.cc:648] HybridClock initialized: now 1790610249483145 us; error 70 us; skew 500 ppm
I20260928 15:44:09.483582 14530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:09 dist-test-slave-j968 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:09.490895 14530 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:09.492570 14530 webserver.cc:460] Webserver started at http://127.13.5.65:35505/ using document root <none> and password file <none>
I20260928 15:44:09.493247 14530 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:09.493333 14530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:09.493625 14530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:09.496734 14530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "53716c10c8e74edd9363679cc9b1e33c"
format_stamp: "Formatted at 2026-09-28 15:44:09 on dist-test-slave-j968"
server_key: "a8004a123911a447c47092593e148cfe"
server_key_iv: "430a75b8850686748bf2b819c656cb34"
server_key_version: "encryptionkey@0"
I20260928 15:44:09.497495 14530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "53716c10c8e74edd9363679cc9b1e33c"
format_stamp: "Formatted at 2026-09-28 15:44:09 on dist-test-slave-j968"
server_key: "a8004a123911a447c47092593e148cfe"
server_key_iv: "430a75b8850686748bf2b819c656cb34"
server_key_version: "encryptionkey@0"
I20260928 15:44:09.502632 14530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:44:09.506075 14545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:09.507434 14530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:44:09.507632 14530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "53716c10c8e74edd9363679cc9b1e33c"
format_stamp: "Formatted at 2026-09-28 15:44:09 on dist-test-slave-j968"
server_key: "a8004a123911a447c47092593e148cfe"
server_key_iv: "430a75b8850686748bf2b819c656cb34"
server_key_version: "encryptionkey@0"
I20260928 15:44:09.507851 14530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:09.522926 14530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:09.527453 14530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:09.527748 14530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:09.530594 14530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:09.530704 14530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:09.530871 14530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:09.530941 14530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:44:09 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610249, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:09.554731 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:09.531037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:51417 (local address 127.13.5.126:43663)
0928 15:44:09.531979 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:09.531983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:09.532774 (+ 791us) server_negotiation.cc:407] Connection header received
0928 15:44:09.534926 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:09.534932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:09.535024 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:09.535153 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:09.537283 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:09.538397 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:09.540881 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:09.541599 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:09.541751 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:09.546881 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:09.546889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:09.546897 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:09.546943 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:09.550961 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:09.552089 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:09.552098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:09.552105 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:09.552299 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:09.552801 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:09.552807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:09.552810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:09.553226 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:09.553388 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:09.554141 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 15:44:09.554368 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":668,"threads_started":1}
I20260928 15:44:09.555968 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:09.531688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43663 (local address 127.13.5.65:51417)
0928 15:44:09.532614 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:09.532650 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:09.534643 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:09.535413 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:09.535422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:09.536170 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:09.537079 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:09.537110 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:09.538548 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:09.538553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:09.540694 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:09.540719 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:09.541845 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:09.541855 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:09.542174 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:09.542922 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:09.542944 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:09.546553 (+ 3609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:09.551146 (+ 4593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:09.551159 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:09.551186 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:09.551896 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:09.552421 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:09.552424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:09.552426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:09.552662 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:09.553363 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:09.553371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:09.553751 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 15:44:09.554021 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 15:44:09.554577 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":462,"threads_started":1}
I20260928 15:44:09.574066 14530 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:41879
I20260928 15:44:09.574162 14665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:41879 every 8 connection(s)
I20260928 15:44:09.575817 14530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:09.576953 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 14530
I20260928 15:44:09.577190 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:09.578024 13333 external_mini_cluster.cc:1428] Setting key 822a6038133b8e6dee5ab873143ea6d4
I20260928 15:44:09.578701 14666 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43663
I20260928 15:44:09.579026 14666 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:09.580170 14666 heartbeater.cc:511] Master 127.13.5.126:43663 requested a full tablet report, sending...
I20260928 15:44:09.582748 14459 ts_manager.cc:194] Registered new tserver with Master: 53716c10c8e74edd9363679cc9b1e33c (127.13.5.65:41879)
I20260928 15:44:09.583074 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:09.564678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43663 (local address 127.13.5.65:39049)
0928 15:44:09.565376 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:09.565404 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:09.565629 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:09.566183 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:09.566189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:09.566695 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:09.567001 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:09.567010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:09.568687 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:09.568698 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:09.570811 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:09.570832 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:09.572014 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:09.572019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:09.572163 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:09.572951 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:09.572976 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:09.574361 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:09.578840 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:09.578849 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:09.578855 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:09.579650 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:09.580332 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:09.580341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:09.580347 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:09.580540 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:09.581443 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:09.581452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:09.581720 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 15:44:09.581897 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 15:44:09.582391 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":389,"threads_started":1}
I20260928 15:44:09.584503 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:09.564700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:39049 (local address 127.13.5.126:43663)
0928 15:44:09.564923 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:09.564929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:09.565519 (+ 590us) server_negotiation.cc:407] Connection header received
0928 15:44:09.565845 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:09.565853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:09.565951 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:09.566094 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:09.567168 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:09.568508 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:09.571015 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:09.571798 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:09.571964 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:09.574601 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:09.574608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:09.574613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:09.574658 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:09.578622 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:09.579948 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:09.579957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:09.579963 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:09.580169 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:09.580706 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:09.580714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:09.580719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:09.581249 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:09.583395 (+ 2146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:09.583567 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 15:44:09.583934 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:44:09.585098 14459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:51417
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:09.623541 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43663
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:09.955207 14670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:09.955621 14670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:09.955789 14670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:09.967654 14670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:09.967809 14670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:09.967967 14670 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:44:09.968088 14670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:09.979854 14670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:43663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:09.981289 14670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:09.982676 14670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:09.996964 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:09.997074 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:09.996982 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:10.001641 14670 server_base.cc:1034] running on GCE node
I20260928 15:44:10.002326 14670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:10.003368 14670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:10.004629 14670 hybrid_clock.cc:648] HybridClock initialized: now 1790610250004553 us; error 55 us; skew 500 ppm
I20260928 15:44:10.004952 14670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:10 dist-test-slave-j968 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:10.008241 14670 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:10.010268 14670 webserver.cc:460] Webserver started at http://127.13.5.66:41993/ using document root <none> and password file <none>
I20260928 15:44:10.011219 14670 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:10.011392 14670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:10.011803 14670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:10.015266 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "ad727a9a96694395b470aae784f185cd"
format_stamp: "Formatted at 2026-09-28 15:44:10 on dist-test-slave-j968"
server_key: "5ac3e283c333bc9bb82824d19c291f16"
server_key_iv: "81c2e2caf92f190b876e362c0dab9a60"
server_key_version: "encryptionkey@0"
I20260928 15:44:10.016194 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "ad727a9a96694395b470aae784f185cd"
format_stamp: "Formatted at 2026-09-28 15:44:10 on dist-test-slave-j968"
server_key: "5ac3e283c333bc9bb82824d19c291f16"
server_key_iv: "81c2e2caf92f190b876e362c0dab9a60"
server_key_version: "encryptionkey@0"
I20260928 15:44:10.021725 14670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:44:10.025365 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:10.026739 14670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 15:44:10.026922 14670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "ad727a9a96694395b470aae784f185cd"
format_stamp: "Formatted at 2026-09-28 15:44:10 on dist-test-slave-j968"
server_key: "5ac3e283c333bc9bb82824d19c291f16"
server_key_iv: "81c2e2caf92f190b876e362c0dab9a60"
server_key_version: "encryptionkey@0"
I20260928 15:44:10.027153 14670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:10.060371 14670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:10.064643 14670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:10.064939 14670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:10.067626 14670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:10.067740 14670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:10.067903 14670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:10.067973 14670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 28 15:44:10 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:10.093824 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.068506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:40803 (local address 127.13.5.126:43663)
0928 15:44:10.068708 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.068712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.070332 (+ 1620us) server_negotiation.cc:407] Connection header received
0928 15:44:10.072894 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.072910 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.073037 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.073131 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:10.075885 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.077128 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.080527 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.081174 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.081304 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.086453 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:10.086463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:10.086471 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:10.086522 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:10.089812 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.090959 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.090966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.090973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.091112 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.091747 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.091755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.091761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.092250 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:10.092423 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.093239 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.093466 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:44:10.095137 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.069290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43663 (local address 127.13.5.66:40803)
0928 15:44:10.070160 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.070200 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.072490 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.073775 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.073784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.074625 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:10.075664 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.075689 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.077382 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.077394 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.080341 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.080364 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.081471 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.081480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.081883 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.082776 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:10.082812 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:10.086188 (+ 3376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:10.089985 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:10.089997 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:10.090025 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:10.090797 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:10.091239 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:10.091246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:10.091251 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:10.091589 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:10.092417 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:10.092425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:10.092832 (+ 407us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.093108 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.093678 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":388,"threads_started":1}
I20260928 15:44:10.120086 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.103306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:48185 (local address 127.13.5.126:43663)
0928 15:44:10.103531 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.103540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.104379 (+ 839us) server_negotiation.cc:407] Connection header received
0928 15:44:10.104588 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.104594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.104674 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.104780 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:10.105712 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.106462 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.107810 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.108406 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.108549 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.111711 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:10.111720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:10.111728 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:10.111784 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:10.116223 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.117501 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.117505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.117508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.117708 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.118305 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.118310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.118314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.118684 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:10.118810 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.119521 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.119700 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:44:10.120568 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.103429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43663 (local address 127.13.5.66:48185)
0928 15:44:10.104270 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.104296 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.104451 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.104806 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.104810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.105270 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:10.105562 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.105571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.106610 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.106615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.107662 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.107674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.108670 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.108680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.108958 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.109837 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:10.109856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:10.111459 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:10.116522 (+ 5063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:10.116529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:10.116533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:10.117291 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:10.117864 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:10.117872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:10.117878 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:10.118150 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:10.118832 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:10.118840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:10.119128 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.119327 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.119884 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":456,"threads_started":1}
I20260928 15:44:10.126047 14670 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:44531
I20260928 15:44:10.126143 14804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:44531 every 8 connection(s)
I20260928 15:44:10.127210 14670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:10.129313 14805 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43663
I20260928 15:44:10.129563 14805 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:10.130249 14805 heartbeater.cc:511] Master 127.13.5.126:43663 requested a full tablet report, sending...
I20260928 15:44:10.131850 14459 ts_manager.cc:194] Registered new tserver with Master: ad727a9a96694395b470aae784f185cd (127.13.5.66:44531)
I20260928 15:44:10.132928 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 14670
I20260928 15:44:10.133131 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:10.133162 14459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:40803
I20260928 15:44:10.133716 13333 external_mini_cluster.cc:1428] Setting key 70e9c8a9e91996b192020efbb603353c
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:10.178831 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43663
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:10.510743 14809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:10.511034 14809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:10.511221 14809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:10.519088 14809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:10.519204 14809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:10.519367 14809 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:44:10.519518 14809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:10.529495 14809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:43663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:10.530818 14809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:10.532456 14809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:10.546092 14815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:10.546171 14814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:10.546124 14817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:10.552358 14809 server_base.cc:1034] running on GCE node
I20260928 15:44:10.553046 14809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:10.554021 14809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:10.555292 14809 hybrid_clock.cc:648] HybridClock initialized: now 1790610250555234 us; error 29 us; skew 500 ppm
I20260928 15:44:10.555722 14809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:10 dist-test-slave-j968 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:10.559052 14809 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:10.560762 14809 webserver.cc:460] Webserver started at http://127.13.5.67:43717/ using document root <none> and password file <none>
I20260928 15:44:10.561548 14809 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:10.561681 14809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:10.562021 14809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:10.565104 14809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "a7731db0083142c98daaa0e04ddd84c7"
format_stamp: "Formatted at 2026-09-28 15:44:10 on dist-test-slave-j968"
server_key: "1bf4673b9c032c71ad054786aa8a01ae"
server_key_iv: "e24b5d1c2a340e00c33349977a417d8d"
server_key_version: "encryptionkey@0"
I20260928 15:44:10.566022 14809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "a7731db0083142c98daaa0e04ddd84c7"
format_stamp: "Formatted at 2026-09-28 15:44:10 on dist-test-slave-j968"
server_key: "1bf4673b9c032c71ad054786aa8a01ae"
server_key_iv: "e24b5d1c2a340e00c33349977a417d8d"
server_key_version: "encryptionkey@0"
I20260928 15:44:10.571434 14809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:44:10.575706 14824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:10.577510 14809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:44:10.577674 14809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "a7731db0083142c98daaa0e04ddd84c7"
format_stamp: "Formatted at 2026-09-28 15:44:10 on dist-test-slave-j968"
server_key: "1bf4673b9c032c71ad054786aa8a01ae"
server_key_iv: "e24b5d1c2a340e00c33349977a417d8d"
server_key_version: "encryptionkey@0"
I20260928 15:44:10.577912 14809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:10.591521 14666 heartbeater.cc:503] Master 127.13.5.126:43663 was elected leader, sending a full tablet report...
I20260928 15:44:10.594676 14809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:10.599316 14809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:10.599608 14809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:10.602514 14809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:10.602631 14809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:10.602767 14809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:10.602839 14809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:44:10 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:10.630569 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.604011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43663 (local address 127.13.5.67:39423)
0928 15:44:10.604889 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.604938 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.607275 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.608494 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.608513 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.609405 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:10.610411 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.610449 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.612313 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.612325 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.614461 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.614482 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.615353 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.615364 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.615565 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.616407 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:10.616440 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:10.620922 (+ 4482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:10.625331 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:10.625342 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:10.625361 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:10.625951 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:10.626539 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:10.626543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:10.626545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:10.626798 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:10.627615 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:10.627622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:10.628078 (+ 456us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.628403 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.628948 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":404,"threads_started":1}
I20260928 15:44:10.631428 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.603189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:39423 (local address 127.13.5.126:43663)
0928 15:44:10.603390 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.603398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.605068 (+ 1670us) server_negotiation.cc:407] Connection header received
0928 15:44:10.607690 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.607697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.607795 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.607921 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:10.610675 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.612105 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.614703 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.615125 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.615223 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.621332 (+ 6109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:10.621357 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:10.621364 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:10.621416 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:10.625166 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.626154 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.626163 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.626169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.626388 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.626941 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.626949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.626955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.627467 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:10.627654 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.630614 (+ 2960us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.631142 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:44:10.651770 14809 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:42179
I20260928 15:44:10.651813 14943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:42179 every 8 connection(s)
I20260928 15:44:10.653149 14809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:10.653327 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.637268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:55843 (local address 127.13.5.126:43663)
0928 15:44:10.637836 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.637844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.638290 (+ 446us) server_negotiation.cc:407] Connection header received
0928 15:44:10.638521 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.638530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.638659 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.638803 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:10.640203 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.641158 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.642962 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.643748 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.643956 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.647008 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:10.647015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:10.647020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:10.647068 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:10.650261 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.651187 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.651194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.651198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.651333 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.651725 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.651732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.651735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.652118 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:10.652284 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.652682 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.652940 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260928 15:44:10.653491 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.637415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43663 (local address 127.13.5.67:55843)
0928 15:44:10.638147 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.638171 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.638351 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.638819 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.638823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.639275 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:10.639966 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.639979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.641427 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.641437 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.642800 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.642813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.644021 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.644034 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.644289 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.645094 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:10.645124 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:10.646749 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:10.650408 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:10.650413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:10.650417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:10.651057 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:10.651448 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:10.651453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:10.651456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:10.651593 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:10.652249 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:10.652255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:10.652443 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.652588 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.652963 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":429,"threads_started":1}
I20260928 15:44:10.656622 14944 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43663
I20260928 15:44:10.656845 14944 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:10.657624 14944 heartbeater.cc:511] Master 127.13.5.126:43663 requested a full tablet report, sending...
I20260928 15:44:10.657728 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 14809
I20260928 15:44:10.657929 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:10.658509 13333 external_mini_cluster.cc:1428] Setting key 31de4d11b629065b872f6dac80a02b84
I20260928 15:44:10.659286 14459 ts_manager.cc:194] Registered new tserver with Master: a7731db0083142c98daaa0e04ddd84c7 (127.13.5.67:42179)
I20260928 15:44:10.660351 14459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:39423
I20260928 15:44:10.661515 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:44:10 dist-test-slave-j968 krb5kdc[14414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:44:10 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:10.708520 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.694453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.13.5.126:43663)
0928 15:44:10.694712 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.694724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.694849 (+ 125us) server_negotiation.cc:407] Connection header received
0928 15:44:10.695002 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.695006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.695075 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.695165 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:10.696871 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.697787 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.699113 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.699521 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.699624 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.703567 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:10.703572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:10.703575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:10.703612 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:10.706175 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.706769 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.706773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.706775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.706896 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:10.707225 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:10.707228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:10.707230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:10.707496 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:10.707689 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.707971 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.708145 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 15:44:10.712965 14458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:10.719810 14458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:10.744127 14957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.732616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:42179 (local address 127.0.0.1:33612)
0928 15:44:10.733434 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.733482 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.733657 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.734926 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.734933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.734964 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:10.735378 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.735391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.737760 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.737768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.740823 (+ 3055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.740843 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.742800 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.742806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.743007 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.743073 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.743215 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.743423 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":40,"thread_start_us":428,"threads_started":1}
I20260928 15:44:10.744566 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.732713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.13.5.67:42179)
0928 15:44:10.734147 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.734158 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.734197 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:44:10.734309 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.734318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.734611 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.734923 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:10.735655 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.737603 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.741008 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.742589 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.742743 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.743006 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.743287 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.743460 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":1083,"threads_started":1}
I20260928 15:44:10.745334 14961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.734635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41879 (local address 127.0.0.1:52448)
0928 15:44:10.736411 (+ 1776us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.736428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.736557 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.737344 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.737349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.737374 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:10.738040 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.738054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.740020 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.740026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.742305 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.742318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.744558 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.744567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.744767 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.744820 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.744902 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.745035 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"mutex_wait_us":3,"thread_start_us":1526,"threads_started":1}
I20260928 15:44:10.746069 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.734392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.13.5.66:44531)
0928 15:44:10.736504 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.736513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.736546 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:44:10.736643 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.736651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.736884 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.737097 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:10.737838 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.738969 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.742803 (+ 3834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.744195 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.744330 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.745013 (+ 683us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.745581 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.745762 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":970,"threads_started":1}
I20260928 15:44:10.746088 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.735167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.13.5.65:41879)
0928 15:44:10.735995 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.736006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.736685 (+ 679us) server_negotiation.cc:407] Connection header received
0928 15:44:10.736787 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.736795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.737005 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.737225 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:10.738246 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.739825 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.742478 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.743931 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.744070 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.745575 (+ 1505us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.745644 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.745798 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":506,"threads_started":1}
I20260928 15:44:10.747298 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.733941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:44531 (local address 127.0.0.1:34786)
0928 15:44:10.734590 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.734608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.734712 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.737109 (+ 2397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.737114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.737146 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:10.737653 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.737663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.739110 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.739118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.742583 (+ 3465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.742598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.744772 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.744781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.745208 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.745238 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.745342 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.747051 (+ 1709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":250,"thread_start_us":342,"threads_started":1}
I20260928 15:44:10.748595 14874 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f (DEFAULT_TABLE table=test-table [id=9700c2ffe6ea4cae8b8d9cc5d0079081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:10.750222 14874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:10.749722 14596 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f (DEFAULT_TABLE table=test-table [id=9700c2ffe6ea4cae8b8d9cc5d0079081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:10.750301 14735 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f (DEFAULT_TABLE table=test-table [id=9700c2ffe6ea4cae8b8d9cc5d0079081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:10.751122 14596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:10.751405 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:10.764016 14963 tablet_bootstrap.cc:492] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Bootstrap starting.
I20260928 15:44:10.766294 14964 tablet_bootstrap.cc:492] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Bootstrap starting.
I20260928 15:44:10.768074 14963 tablet_bootstrap.cc:654] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:10.768144 14965 tablet_bootstrap.cc:492] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Bootstrap starting.
I20260928 15:44:10.769613 14963 log.cc:824] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:10.770567 14964 tablet_bootstrap.cc:654] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:10.771649 14965 tablet_bootstrap.cc:654] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:10.772010 14964 log.cc:824] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:10.772877 14965 log.cc:824] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:10.773880 14963 tablet_bootstrap.cc:492] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: No bootstrap required, opened a new log
I20260928 15:44:10.774188 14963 ts_tablet_manager.cc:1397] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260928 15:44:10.775179 14964 tablet_bootstrap.cc:492] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: No bootstrap required, opened a new log
I20260928 15:44:10.775456 14964 ts_tablet_manager.cc:1397] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:44:10.775563 14965 tablet_bootstrap.cc:492] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: No bootstrap required, opened a new log
I20260928 15:44:10.775795 14965 ts_tablet_manager.cc:1397] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:44:10.783986 14965 raft_consensus.cc:348] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.784394 14965 raft_consensus.cc:374] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:10.784497 14965 raft_consensus.cc:729] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7731db0083142c98daaa0e04ddd84c7, State: Initialized, Role: FOLLOWER
I20260928 15:44:10.785163 14965 consensus_queue.cc:260] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.785588 14963 raft_consensus.cc:348] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.786039 14963 raft_consensus.cc:374] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:10.786147 14963 raft_consensus.cc:729] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad727a9a96694395b470aae784f185cd, State: Initialized, Role: FOLLOWER
I20260928 15:44:10.786602 14944 heartbeater.cc:503] Master 127.13.5.126:43663 was elected leader, sending a full tablet report...
I20260928 15:44:10.786774 14963 consensus_queue.cc:260] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.787060 14965 ts_tablet_manager.cc:1428] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260928 15:44:10.787475 14964 raft_consensus.cc:348] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.787932 14964 raft_consensus.cc:374] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:10.787988 14805 heartbeater.cc:503] Master 127.13.5.126:43663 was elected leader, sending a full tablet report...
I20260928 15:44:10.788043 14964 raft_consensus.cc:729] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53716c10c8e74edd9363679cc9b1e33c, State: Initialized, Role: FOLLOWER
I20260928 15:44:10.788578 14963 ts_tablet_manager.cc:1428] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260928 15:44:10.788627 14964 consensus_queue.cc:260] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.790410 14964 ts_tablet_manager.cc:1428] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260928 15:44:10.843289 14667 tablet.cc:2369] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:10.879284 14806 tablet.cc:2369] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:10.906268 14945 tablet.cc:2369] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:44:10.944669 14971 raft_consensus.cc:482] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:10.944957 14971 raft_consensus.cc:504] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.946656 14971 leader_election.cc:288] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7731db0083142c98daaa0e04ddd84c7 (127.13.5.67:42179), ad727a9a96694395b470aae784f185cd (127.13.5.66:44531)
I20260928 15:44:10.954577 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.947944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:47715 (local address 127.13.5.66:44531)
0928 15:44:10.948179 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.948186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.948378 (+ 192us) server_negotiation.cc:407] Connection header received
0928 15:44:10.948635 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.948640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.948718 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.948820 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:10.949450 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.950514 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.952782 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.953543 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.953711 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.953821 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.954170 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.954314 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 15:44:10.954771 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.946963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:35129 (local address 127.13.5.67:42179)
0928 15:44:10.947150 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.947155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.947692 (+ 537us) server_negotiation.cc:407] Connection header received
0928 15:44:10.948065 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.948071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.948154 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.948219 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:10.949066 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.949943 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.952419 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.953604 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.953729 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.953842 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.954353 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.954515 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:44:10.955078 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.947662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:44531 (local address 127.13.5.65:47715)
0928 15:44:10.948273 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.948287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.948445 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.948839 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.948843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.948872 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:10.949277 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.949288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.950670 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.950680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.952634 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.952644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.953764 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.953770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.953976 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.954025 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.954128 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.954265 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":304,"threads_started":1}
I20260928 15:44:10.955470 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.946863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:42179 (local address 127.13.5.65:35129)
0928 15:44:10.947580 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.947604 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.947823 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.948455 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.948460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.948496 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:10.948911 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.948921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.950080 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.950088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.952246 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.952267 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.953981 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.953987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.954182 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.954211 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.954845 (+ 634us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.954982 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":3,"thread_start_us":363,"threads_started":1}
I20260928 15:44:10.955686 14969 raft_consensus.cc:482] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:10.956025 14969 raft_consensus.cc:504] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.956365 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb59c1f5cb243b6ba7e60c271f3e05f" candidate_uuid: "53716c10c8e74edd9363679cc9b1e33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad727a9a96694395b470aae784f185cd" is_pre_election: true
I20260928 15:44:10.956763 14894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb59c1f5cb243b6ba7e60c271f3e05f" candidate_uuid: "53716c10c8e74edd9363679cc9b1e33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7731db0083142c98daaa0e04ddd84c7" is_pre_election: true
I20260928 15:44:10.956925 14755 raft_consensus.cc:2454] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53716c10c8e74edd9363679cc9b1e33c in term 0.
I20260928 15:44:10.957187 14894 raft_consensus.cc:2454] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53716c10c8e74edd9363679cc9b1e33c in term 0.
I20260928 15:44:10.957887 14547 leader_election.cc:302] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53716c10c8e74edd9363679cc9b1e33c, a7731db0083142c98daaa0e04ddd84c7; no voters:
I20260928 15:44:10.958209 14969 leader_election.cc:288] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad727a9a96694395b470aae784f185cd (127.13.5.66:44531), 53716c10c8e74edd9363679cc9b1e33c (127.13.5.65:41879)
I20260928 15:44:10.958341 14971 raft_consensus.cc:2781] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:10.958479 14971 raft_consensus.cc:482] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:10.958570 14971 raft_consensus.cc:3037] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:10.961764 14971 raft_consensus.cc:504] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.962396 14971 leader_election.cc:288] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [CANDIDATE]: Term 1 election: Requested vote from peers a7731db0083142c98daaa0e04ddd84c7 (127.13.5.67:42179), ad727a9a96694395b470aae784f185cd (127.13.5.66:44531)
I20260928 15:44:10.963616 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb59c1f5cb243b6ba7e60c271f3e05f" candidate_uuid: "53716c10c8e74edd9363679cc9b1e33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad727a9a96694395b470aae784f185cd"
I20260928 15:44:10.963825 14755 raft_consensus.cc:3037] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:10.967314 14755 raft_consensus.cc:2454] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53716c10c8e74edd9363679cc9b1e33c in term 1.
I20260928 15:44:10.967917 14547 leader_election.cc:302] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53716c10c8e74edd9363679cc9b1e33c, ad727a9a96694395b470aae784f185cd; no voters:
I20260928 15:44:10.968324 14971 raft_consensus.cc:2781] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:10.968827 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.958650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:44531 (local address 127.13.5.67:52965)
0928 15:44:10.959321 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.959345 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.959487 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.960033 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.960040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.960075 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:10.960356 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.960369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.962386 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.962395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.965203 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.965228 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.967635 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.967645 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.967963 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.968032 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.968132 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.968381 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515}
I20260928 15:44:10.973970 14894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb59c1f5cb243b6ba7e60c271f3e05f" candidate_uuid: "53716c10c8e74edd9363679cc9b1e33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7731db0083142c98daaa0e04ddd84c7"
I20260928 15:44:10.974262 14971 raft_consensus.cc:686] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 1 LEADER]: Becoming Leader. State: Replica: 53716c10c8e74edd9363679cc9b1e33c, State: Running, Role: LEADER
I20260928 15:44:10.974268 14894 raft_consensus.cc:3037] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:10.974941 14971 consensus_queue.cc:237] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } }
I20260928 15:44:10.975315 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.958831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:52965 (local address 127.13.5.66:44531)
0928 15:44:10.959019 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.959025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.959670 (+ 645us) server_negotiation.cc:407] Connection header received
0928 15:44:10.959753 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.959761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.959857 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.960517 (+ 660us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:10.960715 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.962161 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.965499 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.967374 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.974666 (+ 7292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.974804 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.974871 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.975030 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:44:10.975883 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb59c1f5cb243b6ba7e60c271f3e05f" candidate_uuid: "a7731db0083142c98daaa0e04ddd84c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad727a9a96694395b470aae784f185cd" is_pre_election: true
I20260928 15:44:10.976257 14755 raft_consensus.cc:2379] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a7731db0083142c98daaa0e04ddd84c7 in current term 1: Already voted for candidate 53716c10c8e74edd9363679cc9b1e33c in this term.
I20260928 15:44:10.977768 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.962706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:49965 (local address 127.13.5.65:41879)
0928 15:44:10.962879 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:10.962884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:10.969242 (+ 6358us) server_negotiation.cc:407] Connection header received
0928 15:44:10.969564 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:10.969573 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:10.969699 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:10.969776 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:10.970545 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.972059 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.975136 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.976406 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.976527 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.976628 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:10.977389 (+ 761us) server_negotiation.cc:299] Negotiation successful
0928 15:44:10.977543 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:44:10.977804 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:10.959016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41879 (local address 127.13.5.67:49965)
0928 15:44:10.969123 (+ 10107us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:10.969149 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:10.969371 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:10.970031 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:10.970036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:10.970073 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:10.970334 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.970348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.972296 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.972305 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.974392 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:10.974407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:10.976911 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:10.976917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:10.977174 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:10.977209 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:44:10.977352 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 15:44:10.977487 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9971,"thread_start_us":14316,"threads_started":1}
I20260928 15:44:10.978055 14894 raft_consensus.cc:2454] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53716c10c8e74edd9363679cc9b1e33c in term 1.
I20260928 15:44:10.978886 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb59c1f5cb243b6ba7e60c271f3e05f" candidate_uuid: "a7731db0083142c98daaa0e04ddd84c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53716c10c8e74edd9363679cc9b1e33c" is_pre_election: true
I20260928 15:44:10.980094 14826 leader_election.cc:302] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7731db0083142c98daaa0e04ddd84c7; no voters: 53716c10c8e74edd9363679cc9b1e33c, ad727a9a96694395b470aae784f185cd
I20260928 15:44:10.980572 14969 raft_consensus.cc:2726] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:44:10.982271 14458 catalog_manager.cc:5495] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c reported cstate change: term changed from 0 to 1, leader changed from <none> to 53716c10c8e74edd9363679cc9b1e33c (127.13.5.65). New cstate: current_term: 1 leader_uuid: "53716c10c8e74edd9363679cc9b1e33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } health_report { overall_health: HEALTHY } } }
I20260928 15:44:11.054340 14962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.048672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.13.5.65:41879)
0928 15:44:11.048894 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.048900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.048923 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:44:11.049236 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.049240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.049335 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.049484 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:11.049880 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.050655 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.052030 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.052438 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.052586 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.053103 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:11.053656 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:11.053813 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.053950 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.054048 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:44:11.062785 14755 raft_consensus.cc:1261] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Refusing update from remote peer 53716c10c8e74edd9363679cc9b1e33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:11.062839 14894 raft_consensus.cc:1261] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Refusing update from remote peer 53716c10c8e74edd9363679cc9b1e33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:11.063692 14975 consensus_queue.cc:1035] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:11.064216 14971 consensus_queue.cc:1035] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:11.076538 14978 mvcc.cc:205] Tried to move back new op lower bound from 7334339588343345152 to 7334339588000350208. Current Snapshot: MvccSnapshot[applied={T|T < 7334339588343345152}]
I20260928 15:44:11.097239 14458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:11.097599 14458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:11.101547 14458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27: Sending DeleteTablet for 3 replicas of tablet 5cb59c1f5cb243b6ba7e60c271f3e05f
I20260928 15:44:11.102919 14874 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:11 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:33612
I20260928 15:44:11.102924 14596 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:11 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52448
I20260928 15:44:11.102917 14735 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb59c1f5cb243b6ba7e60c271f3e05f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:11 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:34786
I20260928 15:44:11.107980 14458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:11.108848 14458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:11.116583 14989 tablet_replica.cc:330] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: stopping tablet replica
I20260928 15:44:11.117324 14989 raft_consensus.cc:2229] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:44:11.118178 14989 raft_consensus.cc:2258] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:11.119036 14990 tablet_replica.cc:330] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: stopping tablet replica
I20260928 15:44:11.119549 14990 raft_consensus.cc:2229] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:11.119990 14596 tablet_service.cc:1467] Processing CreateTablet for tablet 507328b4c5f6476681f36d7d5f7b8b77 (DEFAULT_TABLE table=test-table [id=8503afe96d6f4874bf64b39d68132fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:11.120091 14990 raft_consensus.cc:2258] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:11.120424 14873 tablet_service.cc:1467] Processing CreateTablet for tablet 507328b4c5f6476681f36d7d5f7b8b77 (DEFAULT_TABLE table=test-table [id=8503afe96d6f4874bf64b39d68132fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:11.120576 14596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507328b4c5f6476681f36d7d5f7b8b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:11.120980 14873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507328b4c5f6476681f36d7d5f7b8b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:11.122373 14989 ts_tablet_manager.cc:1905] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:11.129576 14735 tablet_service.cc:1467] Processing CreateTablet for tablet 507328b4c5f6476681f36d7d5f7b8b77 (DEFAULT_TABLE table=test-table [id=8503afe96d6f4874bf64b39d68132fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:11.129868 14989 ts_tablet_manager.cc:1918] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:11.130082 14989 log.cc:1196] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/5cb59c1f5cb243b6ba7e60c271f3e05f
I20260928 15:44:11.130681 14989 ts_tablet_manager.cc:1939] T 5cb59c1f5cb243b6ba7e60c271f3e05f P 53716c10c8e74edd9363679cc9b1e33c: Deleting consensus metadata
I20260928 15:44:11.130734 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507328b4c5f6476681f36d7d5f7b8b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:11.132133 14965 tablet_bootstrap.cc:492] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: Bootstrap starting.
I20260928 15:44:11.132934 14990 ts_tablet_manager.cc:1905] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:11.133955 14965 tablet_bootstrap.cc:654] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:11.135012 14964 tablet_bootstrap.cc:492] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: Bootstrap starting.
I20260928 15:44:11.136663 14964 tablet_bootstrap.cc:654] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:11.140134 14965 tablet_bootstrap.cc:492] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: No bootstrap required, opened a new log
I20260928 15:44:11.140300 14965 ts_tablet_manager.cc:1397] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260928 15:44:11.140676 14990 ts_tablet_manager.cc:1918] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:11.140879 14990 log.cc:1196] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/5cb59c1f5cb243b6ba7e60c271f3e05f
I20260928 15:44:11.141175 14965 raft_consensus.cc:348] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.141438 14965 raft_consensus.cc:374] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:11.141474 14990 ts_tablet_manager.cc:1939] T 5cb59c1f5cb243b6ba7e60c271f3e05f P ad727a9a96694395b470aae784f185cd: Deleting consensus metadata
I20260928 15:44:11.141536 14965 raft_consensus.cc:729] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7731db0083142c98daaa0e04ddd84c7, State: Initialized, Role: FOLLOWER
I20260928 15:44:11.141850 14965 consensus_queue.cc:260] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.142342 14988 tablet_replica.cc:330] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: stopping tablet replica
I20260928 15:44:11.142792 14988 raft_consensus.cc:2229] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:11.143474 14444 catalog_manager.cc:4841] TS ad727a9a96694395b470aae784f185cd (127.13.5.66:44531): tablet 5cb59c1f5cb243b6ba7e60c271f3e05f (table test-table [id=9700c2ffe6ea4cae8b8d9cc5d0079081]) successfully deleted
I20260928 15:44:11.143286 14988 raft_consensus.cc:2258] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:11.144701 14963 tablet_bootstrap.cc:492] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: Bootstrap starting.
I20260928 15:44:11.145354 14965 ts_tablet_manager.cc:1428] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 15:44:11.146255 14963 tablet_bootstrap.cc:654] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:11.147231 14444 catalog_manager.cc:4841] TS 53716c10c8e74edd9363679cc9b1e33c (127.13.5.65:41879): tablet 5cb59c1f5cb243b6ba7e60c271f3e05f (table test-table [id=9700c2ffe6ea4cae8b8d9cc5d0079081]) successfully deleted
I20260928 15:44:11.150497 14988 ts_tablet_manager.cc:1905] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:11.156741 14964 tablet_bootstrap.cc:492] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: No bootstrap required, opened a new log
I20260928 15:44:11.156924 14964 ts_tablet_manager.cc:1397] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20260928 15:44:11.157543 14963 tablet_bootstrap.cc:492] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: No bootstrap required, opened a new log
I20260928 15:44:11.157754 14963 ts_tablet_manager.cc:1397] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260928 15:44:11.157862 14964 raft_consensus.cc:348] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.158140 14964 raft_consensus.cc:374] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:11.158250 14964 raft_consensus.cc:729] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53716c10c8e74edd9363679cc9b1e33c, State: Initialized, Role: FOLLOWER
I20260928 15:44:11.158528 14964 consensus_queue.cc:260] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.158878 14963 raft_consensus.cc:348] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.159080 14964 ts_tablet_manager.cc:1428] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 15:44:11.159205 14963 raft_consensus.cc:374] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:11.159368 14963 raft_consensus.cc:729] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad727a9a96694395b470aae784f185cd, State: Initialized, Role: FOLLOWER
I20260928 15:44:11.159595 14988 ts_tablet_manager.cc:1918] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:11.159777 14988 log.cc:1196] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/5cb59c1f5cb243b6ba7e60c271f3e05f
I20260928 15:44:11.159752 14963 consensus_queue.cc:260] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.160362 14988 ts_tablet_manager.cc:1939] T 5cb59c1f5cb243b6ba7e60c271f3e05f P a7731db0083142c98daaa0e04ddd84c7: Deleting consensus metadata
I20260928 15:44:11.160744 14963 ts_tablet_manager.cc:1428] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 15:44:11.162277 14444 catalog_manager.cc:4841] TS a7731db0083142c98daaa0e04ddd84c7 (127.13.5.67:42179): tablet 5cb59c1f5cb243b6ba7e60c271f3e05f (table test-table [id=9700c2ffe6ea4cae8b8d9cc5d0079081]) successfully deleted
I20260928 15:44:11.322431 14983 raft_consensus.cc:482] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:11.322693 14983 raft_consensus.cc:504] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.323608 14983 leader_election.cc:288] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad727a9a96694395b470aae784f185cd (127.13.5.66:44531), a7731db0083142c98daaa0e04ddd84c7 (127.13.5.67:42179)
I20260928 15:44:11.324352 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507328b4c5f6476681f36d7d5f7b8b77" candidate_uuid: "53716c10c8e74edd9363679cc9b1e33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad727a9a96694395b470aae784f185cd" is_pre_election: true
I20260928 15:44:11.324450 14894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507328b4c5f6476681f36d7d5f7b8b77" candidate_uuid: "53716c10c8e74edd9363679cc9b1e33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7731db0083142c98daaa0e04ddd84c7" is_pre_election: true
I20260928 15:44:11.324643 14755 raft_consensus.cc:2454] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53716c10c8e74edd9363679cc9b1e33c in term 0.
I20260928 15:44:11.324734 14894 raft_consensus.cc:2454] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53716c10c8e74edd9363679cc9b1e33c in term 0.
I20260928 15:44:11.325213 14547 leader_election.cc:302] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53716c10c8e74edd9363679cc9b1e33c, a7731db0083142c98daaa0e04ddd84c7; no voters:
I20260928 15:44:11.325515 14983 raft_consensus.cc:2781] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:11.325659 14983 raft_consensus.cc:482] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:11.325800 14983 raft_consensus.cc:3037] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:11.328583 14983 raft_consensus.cc:504] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.329221 14983 leader_election.cc:288] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [CANDIDATE]: Term 1 election: Requested vote from peers ad727a9a96694395b470aae784f185cd (127.13.5.66:44531), a7731db0083142c98daaa0e04ddd84c7 (127.13.5.67:42179)
I20260928 15:44:11.329716 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507328b4c5f6476681f36d7d5f7b8b77" candidate_uuid: "53716c10c8e74edd9363679cc9b1e33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad727a9a96694395b470aae784f185cd"
I20260928 15:44:11.329831 14894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507328b4c5f6476681f36d7d5f7b8b77" candidate_uuid: "53716c10c8e74edd9363679cc9b1e33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7731db0083142c98daaa0e04ddd84c7"
I20260928 15:44:11.329937 14755 raft_consensus.cc:3037] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:11.330065 14894 raft_consensus.cc:3037] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:11.332458 14755 raft_consensus.cc:2454] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53716c10c8e74edd9363679cc9b1e33c in term 1.
I20260928 15:44:11.332926 14547 leader_election.cc:302] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53716c10c8e74edd9363679cc9b1e33c, ad727a9a96694395b470aae784f185cd; no voters:
I20260928 15:44:11.333015 14894 raft_consensus.cc:2454] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53716c10c8e74edd9363679cc9b1e33c in term 1.
I20260928 15:44:11.333482 14983 raft_consensus.cc:2781] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:11.333680 14983 raft_consensus.cc:686] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 1 LEADER]: Becoming Leader. State: Replica: 53716c10c8e74edd9363679cc9b1e33c, State: Running, Role: LEADER
I20260928 15:44:11.334021 14983 consensus_queue.cc:237] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.337056 14459 catalog_manager.cc:5495] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c reported cstate change: term changed from 0 to 1, leader changed from <none> to 53716c10c8e74edd9363679cc9b1e33c (127.13.5.65). New cstate: current_term: 1 leader_uuid: "53716c10c8e74edd9363679cc9b1e33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:44:11 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610248, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:11.422920 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.406378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43663 (local address 127.0.0.1:43878)
0928 15:44:11.407099 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:11.407122 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:11.407297 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:11.407730 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:11.407763 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:11.408745 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:11.409021 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:11.409036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.410234 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.410240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:11.411783 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:11.411800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.412718 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.412726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:11.412908 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.413644 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:11.413668 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:11.416411 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:11.420325 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:11.420336 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:11.420343 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:11.420758 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:11.421206 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:11.421209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:11.421212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:11.421369 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:11.422093 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:11.422099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:11.422263 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 15:44:11.422403 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:44:11.422625 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":336,"threads_started":1}
I20260928 15:44:11.423321 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.406378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.13.5.126:43663)
0928 15:44:11.407427 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.407432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.407451 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:44:11.407498 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.407502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.407558 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.407646 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:11.409196 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.410091 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.412012 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.412536 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.412679 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.416629 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:11.416636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:11.416641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:11.416678 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:11.420182 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.420900 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.420907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.420911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.421085 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.421500 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.421505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.421509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.421924 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:11.422123 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.422585 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.422825 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":799,"threads_started":1}
I20260928 15:44:11.429095 14459 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:43878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:44:11.430541 14459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:11.438303 14596 tablet_service.cc:1467] Processing CreateTablet for tablet 61dad8b3aee54819a70c0f81ac1feb60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9fadf02856d4db187fd9ad764c553e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:44:11.438702 14874 tablet_service.cc:1467] Processing CreateTablet for tablet 61dad8b3aee54819a70c0f81ac1feb60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9fadf02856d4db187fd9ad764c553e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:44:11.439009 14596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61dad8b3aee54819a70c0f81ac1feb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:11.438776 14735 tablet_service.cc:1467] Processing CreateTablet for tablet 61dad8b3aee54819a70c0f81ac1feb60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9fadf02856d4db187fd9ad764c553e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:44:11.439255 14874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61dad8b3aee54819a70c0f81ac1feb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:11.439383 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61dad8b3aee54819a70c0f81ac1feb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:11.446775 14965 tablet_bootstrap.cc:492] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: Bootstrap starting.
I20260928 15:44:11.447211 14963 tablet_bootstrap.cc:492] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: Bootstrap starting.
I20260928 15:44:11.448680 14965 tablet_bootstrap.cc:654] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:11.449002 14963 tablet_bootstrap.cc:654] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:11.451216 14963 tablet_bootstrap.cc:492] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: No bootstrap required, opened a new log
I20260928 15:44:11.451391 14963 ts_tablet_manager.cc:1397] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 15:44:11.452252 14963 raft_consensus.cc:348] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.452515 14963 raft_consensus.cc:374] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:11.452605 14963 raft_consensus.cc:729] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad727a9a96694395b470aae784f185cd, State: Initialized, Role: FOLLOWER
I20260928 15:44:11.453006 14964 tablet_bootstrap.cc:492] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: Bootstrap starting.
I20260928 15:44:11.452904 14963 consensus_queue.cc:260] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.453583 14963 ts_tablet_manager.cc:1428] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:44:11.453495 14970 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.453909 14970 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:11.455014 14964 tablet_bootstrap.cc:654] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:11.457793 14964 tablet_bootstrap.cc:492] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: No bootstrap required, opened a new log
I20260928 15:44:11.457970 14964 ts_tablet_manager.cc:1397] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 15:44:11.458709 14964 raft_consensus.cc:348] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.458971 14964 raft_consensus.cc:374] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:11.459509 14964 raft_consensus.cc:729] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53716c10c8e74edd9363679cc9b1e33c, State: Initialized, Role: FOLLOWER
I20260928 15:44:11.459839 14964 consensus_queue.cc:260] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.460490 14983 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.460796 14983 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:11.461817 14964 ts_tablet_manager.cc:1428] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 15:44:11.463467 14965 tablet_bootstrap.cc:492] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: No bootstrap required, opened a new log
I20260928 15:44:11.463651 14965 ts_tablet_manager.cc:1397] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20260928 15:44:11.464537 14965 raft_consensus.cc:348] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.464802 14965 raft_consensus.cc:374] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:11.464891 14965 raft_consensus.cc:729] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7731db0083142c98daaa0e04ddd84c7, State: Initialized, Role: FOLLOWER
I20260928 15:44:11.465140 14965 consensus_queue.cc:260] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.465698 14965 ts_tablet_manager.cc:1428] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:44:11.465636 14969 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.465909 14969 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:11.509696 14969 raft_consensus.cc:482] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:11.509917 14969 raft_consensus.cc:504] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.510630 14969 leader_election.cc:288] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53716c10c8e74edd9363679cc9b1e33c (127.13.5.65:41879), ad727a9a96694395b470aae784f185cd (127.13.5.66:44531)
I20260928 15:44:11.511426 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61dad8b3aee54819a70c0f81ac1feb60" candidate_uuid: "a7731db0083142c98daaa0e04ddd84c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53716c10c8e74edd9363679cc9b1e33c" is_pre_election: true
I20260928 15:44:11.511404 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61dad8b3aee54819a70c0f81ac1feb60" candidate_uuid: "a7731db0083142c98daaa0e04ddd84c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad727a9a96694395b470aae784f185cd" is_pre_election: true
I20260928 15:44:11.511688 14755 raft_consensus.cc:2454] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7731db0083142c98daaa0e04ddd84c7 in term 0.
I20260928 15:44:11.511773 14616 raft_consensus.cc:2454] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7731db0083142c98daaa0e04ddd84c7 in term 0.
I20260928 15:44:11.512279 14826 leader_election.cc:302] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7731db0083142c98daaa0e04ddd84c7, ad727a9a96694395b470aae784f185cd; no voters:
I20260928 15:44:11.512593 14969 raft_consensus.cc:2781] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:11.512727 14969 raft_consensus.cc:482] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:11.512821 14969 raft_consensus.cc:3037] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:11.516044 14969 raft_consensus.cc:504] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.516732 14969 leader_election.cc:288] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [CANDIDATE]: Term 1 election: Requested vote from peers 53716c10c8e74edd9363679cc9b1e33c (127.13.5.65:41879), ad727a9a96694395b470aae784f185cd (127.13.5.66:44531)
I20260928 15:44:11.517346 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61dad8b3aee54819a70c0f81ac1feb60" candidate_uuid: "a7731db0083142c98daaa0e04ddd84c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53716c10c8e74edd9363679cc9b1e33c"
I20260928 15:44:11.517403 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61dad8b3aee54819a70c0f81ac1feb60" candidate_uuid: "a7731db0083142c98daaa0e04ddd84c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad727a9a96694395b470aae784f185cd"
I20260928 15:44:11.517616 14616 raft_consensus.cc:3037] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:11.517632 14755 raft_consensus.cc:3037] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:11.520535 14616 raft_consensus.cc:2454] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7731db0083142c98daaa0e04ddd84c7 in term 1.
I20260928 15:44:11.520643 14755 raft_consensus.cc:2454] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7731db0083142c98daaa0e04ddd84c7 in term 1.
I20260928 15:44:11.521013 14826 leader_election.cc:302] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53716c10c8e74edd9363679cc9b1e33c, a7731db0083142c98daaa0e04ddd84c7; no voters:
I20260928 15:44:11.521384 14969 raft_consensus.cc:2781] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:11.521788 14969 raft_consensus.cc:686] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [term 1 LEADER]: Becoming Leader. State: Replica: a7731db0083142c98daaa0e04ddd84c7, State: Running, Role: LEADER
I20260928 15:44:11.522217 14969 consensus_queue.cc:237] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } }
I20260928 15:44:11.524340 15005 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: TxnStatusTablet state changed. Reason: New leader a7731db0083142c98daaa0e04ddd84c7. Latest consensus state: current_term: 1 leader_uuid: "a7731db0083142c98daaa0e04ddd84c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.524732 15005 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:44:11.525636 15007 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:44:11.525863 15007 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:44:11.526156 14458 catalog_manager.cc:5495] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7731db0083142c98daaa0e04ddd84c7 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "a7731db0083142c98daaa0e04ddd84c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } health_report { overall_health: HEALTHY } } }
I20260928 15:44:11.549078 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.543003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:42179 (local address 127.0.0.1:33616)
0928 15:44:11.543325 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:11.543346 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:11.543522 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:11.544114 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:11.544119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:11.544131 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:44:11.544325 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:11.544334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.545463 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.545469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:11.546438 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:11.546449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.547247 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.547253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:11.547449 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.547504 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:44:11.548507 (+ 1003us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:44:11.548546 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:44:11.548618 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:44:11.548814 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260928 15:44:11.549837 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.542685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.13.5.67:42179)
0928 15:44:11.543317 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.543323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.543427 (+ 104us) server_negotiation.cc:407] Connection header received
0928 15:44:11.543734 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.543740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.543918 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.544094 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:11.544488 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.545313 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.546591 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.547085 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.547239 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.547718 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:11.548343 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:11.549367 (+ 1024us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.549447 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.549559 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":358,"threads_started":1}
I20260928 15:44:11.554077 14755 raft_consensus.cc:1261] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Refusing update from remote peer a7731db0083142c98daaa0e04ddd84c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:11.554601 14616 raft_consensus.cc:1261] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c [term 1 FOLLOWER]: Refusing update from remote peer a7731db0083142c98daaa0e04ddd84c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:11.555167 15005 consensus_queue.cc:1035] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:11.555641 14969 consensus_queue.cc:1035] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:11.559459 14970 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: TxnStatusTablet state changed. Reason: New leader a7731db0083142c98daaa0e04ddd84c7. Latest consensus state: current_term: 1 leader_uuid: "a7731db0083142c98daaa0e04ddd84c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.559759 14970 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:11.561430 14980 mvcc.cc:205] Tried to move back new op lower bound from 7334339590357463040 to 7334339590239039488. Current Snapshot: MvccSnapshot[applied={T|T < 7334339590357463040}]
I20260928 15:44:11.564043 14970 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7731db0083142c98daaa0e04ddd84c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.564308 14970 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P ad727a9a96694395b470aae784f185cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:11.570917 15013 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7731db0083142c98daaa0e04ddd84c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.571399 15013 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:44:11.572410 14983 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: TxnStatusTablet state changed. Reason: New leader a7731db0083142c98daaa0e04ddd84c7. Latest consensus state: current_term: 1 leader_uuid: "a7731db0083142c98daaa0e04ddd84c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.572780 14983 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:11.585758 14969 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7731db0083142c98daaa0e04ddd84c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.586104 14969 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P a7731db0083142c98daaa0e04ddd84c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:44:11.603205 14983 tablet_replica.cc:439] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7731db0083142c98daaa0e04ddd84c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53716c10c8e74edd9363679cc9b1e33c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41879 } } peers { permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 } } peers { permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 } } }
I20260928 15:44:11.603562 14983 tablet_replica.cc:442] T 61dad8b3aee54819a70c0f81ac1feb60 P 53716c10c8e74edd9363679cc9b1e33c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:11.616971 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.610118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:60425 (local address 127.13.5.67:42179)
0928 15:44:11.610312 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.610320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.611146 (+ 826us) server_negotiation.cc:407] Connection header received
0928 15:44:11.611415 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.611421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.611549 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.611646 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:11.612346 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.613299 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.614841 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.615316 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.615487 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.616095 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:11.616242 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:11.616382 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.616510 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.616598 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:44:11.617069 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.610271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:42179 (local address 127.13.5.65:60425)
0928 15:44:11.611024 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:11.611040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:11.611253 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:11.611675 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:11.611732 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:11.611746 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:44:11.612100 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:11.612110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.613450 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.613458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:11.614684 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:11.614697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.615562 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.615570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:11.615800 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.615860 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:44:11.616392 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:44:11.616406 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:44:11.616490 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:44:11.616638 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":417,"threads_started":1}
I20260928 15:44:11.628964 14894 raft_consensus.cc:1261] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Refusing update from remote peer 53716c10c8e74edd9363679cc9b1e33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:11.629103 14755 raft_consensus.cc:1261] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Refusing update from remote peer 53716c10c8e74edd9363679cc9b1e33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:11.629715 14975 consensus_queue.cc:1035] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad727a9a96694395b470aae784f185cd" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:11.630308 14975 consensus_queue.cc:1035] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7731db0083142c98daaa0e04ddd84c7" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:11.720580 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.692055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:34673 (local address 127.13.5.65:41879)
0928 15:44:11.708323 (+ 16268us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.708330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.708353 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:44:11.708446 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.708452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.708549 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.708659 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:11.713726 (+ 5067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.715244 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.717947 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.718635 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.718787 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.719483 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:11.719665 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:11.719765 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.720138 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.720232 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16121,"thread_start_us":16159,"threads_started":1}
I20260928 15:44:11.720641 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.692070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41879 (local address 127.13.5.67:34673)
0928 15:44:11.702368 (+ 10298us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:11.702404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:11.706424 (+ 4020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:11.708718 (+ 2294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:11.708723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:11.708737 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:44:11.712134 (+ 3397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:11.712155 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.715461 (+ 3306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.715468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:11.717770 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:11.717789 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.718886 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.718900 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:11.719109 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.719202 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:44:11.719941 (+ 739us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:44:11.719955 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:44:11.720081 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 15:44:11.720234 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10061,"thread_start_us":9891,"threads_started":1}
I20260928 15:44:11.770845 14459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:11.771139 14459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:11.774525 14459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a515e4ed8e7443389fe117a136dda27: Sending DeleteTablet for 3 replicas of tablet 507328b4c5f6476681f36d7d5f7b8b77
I20260928 15:44:11.775592 14735 tablet_service.cc:1514] Processing DeleteTablet for tablet 507328b4c5f6476681f36d7d5f7b8b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:11 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:34786
I20260928 15:44:11.775768 14596 tablet_service.cc:1514] Processing DeleteTablet for tablet 507328b4c5f6476681f36d7d5f7b8b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:11 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52448
I20260928 15:44:11.776340 14874 tablet_service.cc:1514] Processing DeleteTablet for tablet 507328b4c5f6476681f36d7d5f7b8b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:11 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:33612
I20260928 15:44:11.777232 15026 tablet_replica.cc:330] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: stopping tablet replica
I20260928 15:44:11.777462 15026 raft_consensus.cc:2229] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:11.777743 15026 raft_consensus.cc:2258] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:11.778995 15026 ts_tablet_manager.cc:1905] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:11.787077 15026 ts_tablet_manager.cc:1918] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:11.787233 15026 log.cc:1196] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/507328b4c5f6476681f36d7d5f7b8b77
I20260928 15:44:11.787707 15026 ts_tablet_manager.cc:1939] T 507328b4c5f6476681f36d7d5f7b8b77 P a7731db0083142c98daaa0e04ddd84c7: Deleting consensus metadata
I20260928 15:44:11.789101 14444 catalog_manager.cc:4841] TS a7731db0083142c98daaa0e04ddd84c7 (127.13.5.67:42179): tablet 507328b4c5f6476681f36d7d5f7b8b77 (table test-table [id=8503afe96d6f4874bf64b39d68132fe5]) successfully deleted
I20260928 15:44:11.792321 15025 tablet_replica.cc:330] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: stopping tablet replica
I20260928 15:44:11.792589 15025 raft_consensus.cc:2229] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:44:11.793058 15024 tablet_replica.cc:330] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: stopping tablet replica
I20260928 15:44:11.793118 15025 raft_consensus.cc:2258] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:11.793418 15024 raft_consensus.cc:2229] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:11.793752 15024 raft_consensus.cc:2258] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:11.794258 15025 ts_tablet_manager.cc:1905] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:11.796541 15024 ts_tablet_manager.cc:1905] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:11.802649 15025 ts_tablet_manager.cc:1918] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:11.802803 15024 ts_tablet_manager.cc:1918] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:11.802932 15025 log.cc:1196] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/507328b4c5f6476681f36d7d5f7b8b77
I20260928 15:44:11.802964 15024 log.cc:1196] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/507328b4c5f6476681f36d7d5f7b8b77
I20260928 15:44:11.803529 15024 ts_tablet_manager.cc:1939] T 507328b4c5f6476681f36d7d5f7b8b77 P ad727a9a96694395b470aae784f185cd: Deleting consensus metadata
I20260928 15:44:11.803531 15025 ts_tablet_manager.cc:1939] T 507328b4c5f6476681f36d7d5f7b8b77 P 53716c10c8e74edd9363679cc9b1e33c: Deleting consensus metadata
I20260928 15:44:11.804715 14444 catalog_manager.cc:4841] TS 53716c10c8e74edd9363679cc9b1e33c (127.13.5.65:41879): tablet 507328b4c5f6476681f36d7d5f7b8b77 (table test-table [id=8503afe96d6f4874bf64b39d68132fe5]) successfully deleted
I20260928 15:44:11.805063 14444 catalog_manager.cc:4841] TS ad727a9a96694395b470aae784f185cd (127.13.5.66:44531): tablet 507328b4c5f6476681f36d7d5f7b8b77 (table test-table [id=8503afe96d6f4874bf64b39d68132fe5]) successfully deleted
Sep 28 15:44:11 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:11.816633 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.786021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.13.5.65:41879)
0928 15:44:11.787011 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.787021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.795211 (+ 8190us) server_negotiation.cc:407] Connection header received
0928 15:44:11.795456 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.795461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.795557 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.795654 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:11.801777 (+ 6123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.802915 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.805054 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.805762 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.805903 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.810220 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:11.810239 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:11.810267 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:11.810315 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:11.813748 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.814602 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.814609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.814611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.814746 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.815150 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.815153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.815156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.815512 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:11.815721 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.816119 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.816288 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792}
W20260928 15:44:11.817411 14661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52462
I20260928 15:44:11.834671 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.820561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.13.5.126:43663)
0928 15:44:11.820804 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.820812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.821027 (+ 215us) server_negotiation.cc:407] Connection header received
0928 15:44:11.821160 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.821166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.821264 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.821392 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:11.822619 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.823791 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.825095 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.825756 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.825882 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.828402 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:11.828410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:11.828415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:11.828456 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:11.831917 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.832557 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.832562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.832566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.832730 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.833110 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.833116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.833119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.833525 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:11.833777 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.834082 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.834326 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260928 15:44:11.835376 14459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43894
I20260928 15:44:11.837808 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14530
I20260928 15:44:11.851416 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.838730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.13.5.65:41879)
0928 15:44:11.838940 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.838949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.839231 (+ 282us) server_negotiation.cc:407] Connection header received
0928 15:44:11.839422 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.839428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.839510 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.839612 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:11.840904 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.841665 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.842879 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.843309 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.843448 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.845767 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:11.845774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:11.845779 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:11.845822 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:11.848657 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.849345 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.849352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.849356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.849564 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.849980 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.849985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.849988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.850267 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:11.850542 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.850832 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.851062 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 15:44:11.852082 14661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52478
W20260928 15:44:11.852663 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14530: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:11.852857 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 14530
I20260928 15:44:11.883371 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14670
Sep 28 15:44:11 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:44:11.899649 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.884279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.13.5.66:44531)
0928 15:44:11.884835 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.884839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.884855 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:44:11.884911 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.884914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.884977 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.885106 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:11.886339 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.887117 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.888569 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.889027 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.889165 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.893483 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:11.893496 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:11.893518 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:11.893553 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:11.896646 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.897388 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.897394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.897397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.897536 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.897986 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.897992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.897995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.898410 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:11.898732 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.899085 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.899315 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":254,"threads_started":1}
W20260928 15:44:11.900368 14800 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34790
W20260928 15:44:11.901051 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:11.901226 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 14670
I20260928 15:44:11.921052 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14809
Sep 28 15:44:11 dist-test-slave-j968 krb5kdc[14414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:44:11.939079 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:11.922036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.13.5.67:42179)
0928 15:44:11.922245 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:11.922254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:11.922282 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:44:11.922381 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:11.922386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:11.922521 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:11.922662 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:11.923888 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.925092 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.926310 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:11.927020 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:11.927172 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:11.931643 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:11.931656 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:11.931679 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:11.931737 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:11.936125 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.936836 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.936846 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.936850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.937018 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:11.937378 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:11.937382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:11.937386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:11.937883 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:11.938210 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:11.938389 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 15:44:11.938653 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 15:44:11.940181 14939 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33626
W20260928 15:44:11.940899 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14809: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:11.941049 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 14809
I20260928 15:44:11.960966 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 14427
I20260928 15:44:11.962248 14499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:12.077953 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 14427
2026-09-28T15:44:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4960 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:12 dist-test-slave-j968 krb5kdc[15036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:12 dist-test-slave-j968 krb5kdc[15036](info): set up 2 sockets
Sep 28 15:44:12 dist-test-slave-j968 krb5kdc[15036](info): commencing operation
krb5kdc: starting...
W20260928 15:44:13.174752 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:13 dist-test-slave-j968 krb5kdc[15036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:13.368590 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:33683
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:33683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:13.700924 15049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:13.701370 15049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:13.701658 15049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:13.709048 15049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:13.709160 15049 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:44:13.709239 15049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:13.709331 15049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:13.709426 15049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:13.709517 15049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:13.720301 15049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:33683
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:33683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:13.721815 15049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:13.723287 15049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:13.736044 15054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:13.736079 15057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:13.736116 15055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:13.737031 15049 server_base.cc:1034] running on GCE node
I20260928 15:44:13.737869 15049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:13.739432 15049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:13.740705 15049 hybrid_clock.cc:648] HybridClock initialized: now 1790610253740636 us; error 43 us; skew 500 ppm
I20260928 15:44:13.741065 15049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:13 dist-test-slave-j968 krb5kdc[15036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610253, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:13.744516 15049 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:13.746132 15049 webserver.cc:460] Webserver started at http://127.13.5.126:35191/ using document root <none> and password file <none>
I20260928 15:44:13.746817 15049 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:13.746904 15049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:13.747174 15049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:13.750195 15049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "d5622c97ed6e4bfc9c977d3b00d23798"
format_stamp: "Formatted at 2026-09-28 15:44:13 on dist-test-slave-j968"
server_key: "6e7febde4e7976468a4e6909aee5ee85"
server_key_iv: "c0f51863396878de7b79b839132a4ab9"
server_key_version: "encryptionkey@0"
I20260928 15:44:13.750929 15049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "d5622c97ed6e4bfc9c977d3b00d23798"
format_stamp: "Formatted at 2026-09-28 15:44:13 on dist-test-slave-j968"
server_key: "6e7febde4e7976468a4e6909aee5ee85"
server_key_iv: "c0f51863396878de7b79b839132a4ab9"
server_key_version: "encryptionkey@0"
I20260928 15:44:13.756258 15049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:44:13.759799 15064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:13.761364 15049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:13.761547 15049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "d5622c97ed6e4bfc9c977d3b00d23798"
format_stamp: "Formatted at 2026-09-28 15:44:13 on dist-test-slave-j968"
server_key: "6e7febde4e7976468a4e6909aee5ee85"
server_key_iv: "c0f51863396878de7b79b839132a4ab9"
server_key_version: "encryptionkey@0"
I20260928 15:44:13.761750 15049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:13.778132 15049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:13.782913 15049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:13.783146 15049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:13.805330 15049 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:33683
I20260928 15:44:13.805337 15125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:33683 every 8 connection(s)
I20260928 15:44:13.806888 15049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:13.809259 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 15049
I20260928 15:44:13.809512 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:13.810433 13333 external_mini_cluster.cc:1428] Setting key 4455c1f464535c6ca064432384cfc4af
I20260928 15:44:13.812330 15126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:44:13 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:13.828249 15126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: Bootstrap starting.
I20260928 15:44:13.832075 15126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:13.833250 15126 log.cc:824] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:13.835804 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:13.812728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.13.5.126:33683)
0928 15:44:13.813530 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:13.813543 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:13.813590 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:44:13.815266 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:13.815288 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:13.815728 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:13.816162 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:13.818135 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:13.819597 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:13.821609 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:13.822297 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:13.822492 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:13.827628 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:13.827637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:13.827654 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:13.827700 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:13.830936 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:13.831698 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:13.831706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:13.831712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:13.831938 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:13.832368 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:13.832375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:13.832382 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:13.832942 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:13.833245 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:13.833808 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 15:44:13.834306 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":385,"threads_started":1}
I20260928 15:44:13.836813 15126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: No bootstrap required, opened a new log
I20260928 15:44:13.844990 15126 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5622c97ed6e4bfc9c977d3b00d23798" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33683 } }
I20260928 15:44:13.845407 15126 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:13.845521 15126 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5622c97ed6e4bfc9c977d3b00d23798, State: Initialized, Role: FOLLOWER
I20260928 15:44:13.846182 15126 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5622c97ed6e4bfc9c977d3b00d23798" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33683 } }
I20260928 15:44:13.846449 15126 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:13.846601 15126 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:13.846796 15126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:13.849258 15126 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5622c97ed6e4bfc9c977d3b00d23798" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33683 } }
I20260928 15:44:13.849817 15126 leader_election.cc:302] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5622c97ed6e4bfc9c977d3b00d23798; no voters:
I20260928 15:44:13.850431 15126 leader_election.cc:288] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:13.850585 15131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:13.851328 15131 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [term 1 LEADER]: Becoming Leader. State: Replica: d5622c97ed6e4bfc9c977d3b00d23798, State: Running, Role: LEADER
I20260928 15:44:13.852077 15131 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5622c97ed6e4bfc9c977d3b00d23798" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33683 } }
I20260928 15:44:13.852881 15126 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:13.855041 15132 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5622c97ed6e4bfc9c977d3b00d23798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5622c97ed6e4bfc9c977d3b00d23798" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33683 } } }
I20260928 15:44:13.855400 15132 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:13.855957 15133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5622c97ed6e4bfc9c977d3b00d23798. Latest consensus state: current_term: 1 leader_uuid: "d5622c97ed6e4bfc9c977d3b00d23798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5622c97ed6e4bfc9c977d3b00d23798" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33683 } } }
I20260928 15:44:13.856225 15133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:13.860436 15138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:13.866369 15138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:13.876201 15138 catalog_manager.cc:1347] Generated new cluster ID: 93e6fe995c974435997a6c3709782509
I20260928 15:44:13.876340 15138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:13.891124 15138 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:13.892946 15138 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:13.902567 15138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: Generated new TSK 0
I20260928 15:44:13.903479 15138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:13.951953 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33683
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:14.287540 15152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:14.287873 15152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:14.288079 15152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:14.295465 15152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:14.295617 15152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:14.295763 15152 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:44:14.295851 15152 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:44:14.296007 15152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:14.305750 15152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:14.307121 15152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:14.308521 15152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:14.321560 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:14.321623 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:14.321655 15160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:14.322325 15152 server_base.cc:1034] running on GCE node
I20260928 15:44:14.322983 15152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:14.323992 15152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:14.325225 15152 hybrid_clock.cc:648] HybridClock initialized: now 1790610254325169 us; error 49 us; skew 500 ppm
I20260928 15:44:14.325574 15152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:14 dist-test-slave-j968 krb5kdc[15036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:14.328742 15152 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:14.330212 15152 webserver.cc:460] Webserver started at http://127.13.5.65:32855/ using document root <none> and password file <none>
I20260928 15:44:14.330976 15152 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:14.331110 15152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:14.331435 15152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:14.334597 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "67e6e39328174accadf9d1990a77d42f"
format_stamp: "Formatted at 2026-09-28 15:44:14 on dist-test-slave-j968"
server_key: "e362169cd0312b5df3f92f831f44862d"
server_key_iv: "78f8020e25fd3166d60bcac34d442347"
server_key_version: "encryptionkey@0"
I20260928 15:44:14.335387 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "67e6e39328174accadf9d1990a77d42f"
format_stamp: "Formatted at 2026-09-28 15:44:14 on dist-test-slave-j968"
server_key: "e362169cd0312b5df3f92f831f44862d"
server_key_iv: "78f8020e25fd3166d60bcac34d442347"
server_key_version: "encryptionkey@0"
I20260928 15:44:14.340521 15152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 15:44:14.343977 15167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:14.345325 15152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:14.345494 15152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "67e6e39328174accadf9d1990a77d42f"
format_stamp: "Formatted at 2026-09-28 15:44:14 on dist-test-slave-j968"
server_key: "e362169cd0312b5df3f92f831f44862d"
server_key_iv: "78f8020e25fd3166d60bcac34d442347"
server_key_version: "encryptionkey@0"
I20260928 15:44:14.345695 15152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:14.385118 15152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:14.389279 15152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:14.389577 15152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:14.392408 15152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:14.392530 15152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:14.392642 15152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:14.392729 15152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:44:14 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:14.417632 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:14.393054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:33741 (local address 127.13.5.126:33683)
0928 15:44:14.395705 (+ 2651us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:14.395714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:14.395741 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:44:14.397675 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:14.397681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:14.397770 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:14.397907 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:14.400946 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.401917 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.404438 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.405082 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.405223 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:14.410450 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:14.410455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:14.410458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:14.410492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:14.413564 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:14.414458 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:14.414467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:14.414471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:14.414652 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:14.415116 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:14.415125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:14.415130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:14.415656 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:14.415842 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:14.416842 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0928 15:44:14.417132 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2509,"thread_start_us":2382,"threads_started":1}
I20260928 15:44:14.418776 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:14.393788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33683 (local address 127.13.5.65:33741)
0928 15:44:14.394911 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:14.394975 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:14.397315 (+ 2340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:14.398404 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:14.398421 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:14.399459 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:14.400689 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:14.400723 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.402161 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.402170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:14.404286 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:14.404298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.405334 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.405342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:14.405655 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:14.406470 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:14.406501 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:14.410161 (+ 3660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:14.413709 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:14.413718 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:14.413736 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:14.414288 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:14.414783 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:14.414788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:14.414792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:14.414984 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:14.415812 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:14.415822 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:14.416340 (+ 518us) client_negotiation.cc:769] Sending connection context
0928 15:44:14.416718 (+ 378us) client_negotiation.cc:240] Negotiation successful
0928 15:44:14.417416 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":446,"threads_started":1}
I20260928 15:44:14.440544 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:14.425484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33683 (local address 127.13.5.65:34005)
0928 15:44:14.426101 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:14.426127 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:14.426294 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:14.426717 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:14.426722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:14.427154 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:14.427431 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:14.427439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.429032 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.429040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:14.430745 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:14.430756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.431497 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.431503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:14.431668 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:14.432339 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:14.432375 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:14.433541 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:14.437188 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:14.437194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:14.437198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:14.437823 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:14.438349 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:14.438358 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:14.438362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:14.438505 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:14.439188 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:14.439193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:14.439439 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 15:44:14.439654 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 15:44:14.440074 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":336,"threads_started":1}
I20260928 15:44:14.441260 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:14.425374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:34005 (local address 127.13.5.126:33683)
0928 15:44:14.425616 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:14.425625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:14.426226 (+ 601us) server_negotiation.cc:407] Connection header received
0928 15:44:14.426448 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:14.426457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:14.426559 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:14.426667 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:14.427570 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.428877 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.430892 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.431311 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.431453 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:14.433805 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:14.433813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:14.433820 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:14.433862 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:14.437060 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:14.438021 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:14.438028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:14.438034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:14.438212 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:14.438621 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:14.438628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:14.438631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:14.439044 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:14.439210 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:14.440864 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0928 15:44:14.441042 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 15:44:14.444831 15152 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:34285
I20260928 15:44:14.444855 15287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:34285 every 8 connection(s)
I20260928 15:44:14.446220 15152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:14.448230 15288 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33683
I20260928 15:44:14.448477 15288 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:14.449270 15288 heartbeater.cc:511] Master 127.13.5.126:33683 requested a full tablet report, sending...
I20260928 15:44:14.451301 15081 ts_manager.cc:194] Registered new tserver with Master: 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285)
I20260928 15:44:14.452680 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 15152
I20260928 15:44:14.452893 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:14.452937 15081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:33741
I20260928 15:44:14.453660 13333 external_mini_cluster.cc:1428] Setting key c9483cb6fa1b0177d9d305a9356eac07
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:14.495064 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33683
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:14.825984 15292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:14.826296 15292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:14.826401 15292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:14.833943 15292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:14.834064 15292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:14.834177 15292 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:44:14.834231 15292 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:44:14.834292 15292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:14.844476 15292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:14.845774 15292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:14.847070 15292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:14.860558 15298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:14.860523 15297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:14.860615 15300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:14.861218 15292 server_base.cc:1034] running on GCE node
I20260928 15:44:14.861832 15292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:14.862716 15292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:14.863971 15292 hybrid_clock.cc:648] HybridClock initialized: now 1790610254863922 us; error 43 us; skew 500 ppm
I20260928 15:44:14.864351 15292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:14 dist-test-slave-j968 krb5kdc[15036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:14.871348 15292 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:14.872997 15292 webserver.cc:460] Webserver started at http://127.13.5.66:32895/ using document root <none> and password file <none>
I20260928 15:44:14.873715 15292 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:14.873863 15292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:14.874195 15292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:14.877265 15292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "0c622bb34dcf4baaa83492150ca5044d"
format_stamp: "Formatted at 2026-09-28 15:44:14 on dist-test-slave-j968"
server_key: "618c556f2f0094b0efcf18ff9a163320"
server_key_iv: "0167013921ff8f0effa1facd6849005b"
server_key_version: "encryptionkey@0"
I20260928 15:44:14.878044 15292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "0c622bb34dcf4baaa83492150ca5044d"
format_stamp: "Formatted at 2026-09-28 15:44:14 on dist-test-slave-j968"
server_key: "618c556f2f0094b0efcf18ff9a163320"
server_key_iv: "0167013921ff8f0effa1facd6849005b"
server_key_version: "encryptionkey@0"
I20260928 15:44:14.883543 15292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:44:14.887117 15307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:14.888577 15292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:14.888762 15292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "0c622bb34dcf4baaa83492150ca5044d"
format_stamp: "Formatted at 2026-09-28 15:44:14 on dist-test-slave-j968"
server_key: "618c556f2f0094b0efcf18ff9a163320"
server_key_iv: "0167013921ff8f0effa1facd6849005b"
server_key_version: "encryptionkey@0"
I20260928 15:44:14.888980 15292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:14.907951 15292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:14.912093 15292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:14.912397 15292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:14.915215 15292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:14.915359 15292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:14.915484 15292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:14.915570 15292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:44:14 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610254, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:14.942816 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:14.915658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:50305 (local address 127.13.5.126:33683)
0928 15:44:14.915925 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:14.915934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:14.917530 (+ 1596us) server_negotiation.cc:407] Connection header received
0928 15:44:14.920349 (+ 2819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:14.920356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:14.920457 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:14.920571 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:14.923234 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.924610 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.928172 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.929172 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.929323 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:14.934633 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:14.934644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:14.934652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:14.934711 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:14.938654 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:14.939916 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:14.939925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:14.939931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:14.940098 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:14.940745 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:14.940752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:14.940760 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:14.941227 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:14.941405 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:14.942238 (+ 833us) server_negotiation.cc:299] Negotiation successful
0928 15:44:14.942461 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:44:14.944036 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:14.916373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33683 (local address 127.13.5.66:50305)
0928 15:44:14.917356 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:14.917395 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:14.920049 (+ 2654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:14.920941 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:14.920958 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:14.921953 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:14.922943 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:14.922993 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.924872 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.924881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:14.927915 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:14.927938 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.929789 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.929799 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:14.930195 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:14.931141 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:14.931188 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:14.934354 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:14.938860 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:14.938870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:14.938891 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:14.939688 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:14.940240 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:14.940248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:14.940253 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:14.940608 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:14.941384 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:14.941396 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:14.941864 (+ 468us) client_negotiation.cc:769] Sending connection context
0928 15:44:14.942148 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 15:44:14.942711 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":464,"threads_started":1}
I20260928 15:44:14.965279 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:14.950517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:34467 (local address 127.13.5.126:33683)
0928 15:44:14.950764 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:14.950773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:14.951350 (+ 577us) server_negotiation.cc:407] Connection header received
0928 15:44:14.951491 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:14.951500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:14.951613 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:14.951741 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:14.953243 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.954410 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.956483 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.956949 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.957086 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:14.958938 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:14.958946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:14.958956 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:14.958987 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:14.962085 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:14.963145 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:14.963148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:14.963151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:14.963272 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:14.963691 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:14.963694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:14.963696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:14.964079 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:14.964262 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:14.964739 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 15:44:14.964949 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 15:44:14.965451 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:14.950612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33683 (local address 127.13.5.66:34467)
0928 15:44:14.951242 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:14.951267 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:14.951373 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:14.951758 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:14.951761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:14.952434 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:14.953030 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:14.953051 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.954561 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.954568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:14.956332 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:14.956345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:14.957152 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:14.957159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:14.957312 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:14.958027 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:14.958045 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:14.958757 (+ 712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:14.962301 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:14.962306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:14.962311 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:14.963023 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:14.963388 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:14.963395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:14.963399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:14.963577 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:14.964236 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:14.964241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:14.964442 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 15:44:14.964593 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 15:44:14.964924 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":331,"threads_started":1}
I20260928 15:44:14.978379 15292 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43349
I20260928 15:44:14.978415 15426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43349 every 8 connection(s)
I20260928 15:44:14.979612 15292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:14.981889 15427 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33683
I20260928 15:44:14.982085 15427 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:14.982718 15427 heartbeater.cc:511] Master 127.13.5.126:33683 requested a full tablet report, sending...
I20260928 15:44:14.984073 15081 ts_manager.cc:194] Registered new tserver with Master: 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349)
I20260928 15:44:14.984915 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 15292
I20260928 15:44:14.984956 15081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:50305
I20260928 15:44:14.985072 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:14.985631 13333 external_mini_cluster.cc:1428] Setting key 4ba67f45052abe9ac5e532d5b03c190a
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:15.027732 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33683
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:15.356415 15431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:15.356735 15431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:15.356914 15431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:15.364482 15431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:15.364630 15431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:15.364795 15431 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:44:15.364881 15431 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:44:15.365010 15431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:15.375313 15431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:33683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:15.376816 15431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:15.378180 15431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:15.392079 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:15.392079 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:15.392217 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:15.393069 15431 server_base.cc:1034] running on GCE node
I20260928 15:44:15.393751 15431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:15.394809 15431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:15.396013 15431 hybrid_clock.cc:648] HybridClock initialized: now 1790610255395963 us; error 62 us; skew 500 ppm
I20260928 15:44:15.396394 15431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:15 dist-test-slave-j968 krb5kdc[15036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610255, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:15.399609 15431 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:15.401218 15431 webserver.cc:460] Webserver started at http://127.13.5.67:40169/ using document root <none> and password file <none>
I20260928 15:44:15.401940 15431 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:15.402073 15431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:15.402413 15431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:15.405575 15431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "ff611e9554fb41c381cea809b2e7fbe0"
format_stamp: "Formatted at 2026-09-28 15:44:15 on dist-test-slave-j968"
server_key: "6c03b04731f07b52f12171e92a41da25"
server_key_iv: "1b9a160db78df6a3e47260064b4d28f8"
server_key_version: "encryptionkey@0"
I20260928 15:44:15.406347 15431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "ff611e9554fb41c381cea809b2e7fbe0"
format_stamp: "Formatted at 2026-09-28 15:44:15 on dist-test-slave-j968"
server_key: "6c03b04731f07b52f12171e92a41da25"
server_key_iv: "1b9a160db78df6a3e47260064b4d28f8"
server_key_version: "encryptionkey@0"
I20260928 15:44:15.411530 15431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:44:15.414968 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:15.416455 15431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:15.416628 15431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "ff611e9554fb41c381cea809b2e7fbe0"
format_stamp: "Formatted at 2026-09-28 15:44:15 on dist-test-slave-j968"
server_key: "6c03b04731f07b52f12171e92a41da25"
server_key_iv: "1b9a160db78df6a3e47260064b4d28f8"
server_key_version: "encryptionkey@0"
I20260928 15:44:15.416826 15431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:15.442448 15431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:15.447796 15431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:15.448140 15431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:15.451320 15431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:15.451453 15431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:15.451592 15431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:15.451704 15431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:15.456133 15288 heartbeater.cc:503] Master 127.13.5.126:33683 was elected leader, sending a full tablet report...
Sep 28 15:44:15 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610255, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:15.477501 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.451785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:47239 (local address 127.13.5.126:33683)
0928 15:44:15.451996 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.452035 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.453576 (+ 1541us) server_negotiation.cc:407] Connection header received
0928 15:44:15.455791 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.455797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.455914 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.456010 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:15.458421 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.459481 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.462191 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.462795 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.462965 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.469085 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:15.469093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:15.469100 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:15.469146 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:15.473054 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:15.474166 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:15.474174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:15.474179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:15.474365 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:15.475070 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:15.475073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:15.475075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:15.475362 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:15.475498 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.476760 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.477054 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:44:15.479496 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.452541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33683 (local address 127.13.5.67:47239)
0928 15:44:15.453401 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:15.453442 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:15.455490 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:15.456527 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:15.456538 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:15.457332 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:15.458236 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.458264 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.459690 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.459694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.462014 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.462032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.463123 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.463135 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.463533 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.464603 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:15.464643 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:15.468662 (+ 4019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:15.473249 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:15.473266 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:15.473288 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:15.474018 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:15.474517 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:15.474526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:15.474532 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:15.474906 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:15.475500 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:15.475514 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:15.476192 (+ 678us) client_negotiation.cc:769] Sending connection context
0928 15:44:15.476615 (+ 423us) client_negotiation.cc:240] Negotiation successful
0928 15:44:15.477449 (+ 834us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":376,"threads_started":1}
I20260928 15:44:15.500515 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.486621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:45175 (local address 127.13.5.126:33683)
0928 15:44:15.486814 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.486819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.487279 (+ 460us) server_negotiation.cc:407] Connection header received
0928 15:44:15.487491 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.487499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.487603 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.487712 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:15.489099 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.490078 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.491418 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.491832 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.491998 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.494479 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:15.494492 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:15.494496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:15.494534 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:15.497771 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:15.498461 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:15.498467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:15.498470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:15.498609 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:15.498970 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:15.498976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:15.498980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:15.499249 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:15.499455 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.499843 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.500124 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:44:15.500701 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.486119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33683 (local address 127.13.5.67:45175)
0928 15:44:15.487170 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:15.487184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:15.487349 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:15.487771 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:15.487776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:15.488444 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:15.488879 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.488897 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.490226 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.490231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.491273 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.491285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.492072 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.492079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.492322 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.493147 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:15.493177 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:15.494283 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:15.497909 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:15.497918 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:15.497922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:15.498336 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:15.498728 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:15.498733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:15.498736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:15.498865 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:15.499405 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:15.499410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:15.499605 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 15:44:15.499750 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 15:44:15.500124 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":818,"threads_started":1}
I20260928 15:44:15.502975 15431 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:33257
I20260928 15:44:15.503053 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:33257 every 8 connection(s)
I20260928 15:44:15.504761 15431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:15.506844 15566 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33683
I20260928 15:44:15.507066 15566 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:15.507637 15566 heartbeater.cc:511] Master 127.13.5.126:33683 requested a full tablet report, sending...
I20260928 15:44:15.508432 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 15431
I20260928 15:44:15.508641 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:15.509229 13333 external_mini_cluster.cc:1428] Setting key 46299a6d1bda5178db0b5bc3006bf00f
I20260928 15:44:15.509186 15081 ts_manager.cc:194] Registered new tserver with Master: ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257)
I20260928 15:44:15.510183 15081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:47239
I20260928 15:44:15.512259 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:44:15 dist-test-slave-j968 krb5kdc[15036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:44:15 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:15.560701 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.545003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.13.5.126:33683)
0928 15:44:15.545187 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.545191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.545701 (+ 510us) server_negotiation.cc:407] Connection header received
0928 15:44:15.545819 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.545823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.545891 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.545989 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:15.547761 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.548672 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.550141 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.550796 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.550937 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.555334 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:15.555339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:15.555342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:15.555387 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:15.558277 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:15.558906 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:15.558911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:15.558913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:15.559060 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:15.559385 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:15.559388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:15.559390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:15.559685 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:15.559957 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.560241 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.560416 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:44:15.565042 15081 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:15.572327 15081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:15.597317 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.585485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:33257 (local address 127.0.0.1:47194)
0928 15:44:15.588011 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:15.588047 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:15.588246 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:15.589367 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:15.589378 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:15.589424 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:15.589960 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.589983 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.591919 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.591923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.594236 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.594248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.595979 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.595985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.596400 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.596460 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 15:44:15.596596 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:44:15.596809 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2218,"mutex_wait_us":53,"thread_start_us":2064,"threads_started":1}
I20260928 15:44:15.597823 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.585664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43349 (local address 127.0.0.1:56012)
0928 15:44:15.587990 (+ 2326us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:15.588047 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:15.588246 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:15.589411 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:15.589415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:15.589441 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:15.590015 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.590039 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.591714 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.591722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.594608 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.594622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.596726 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.596738 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.596897 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.596969 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 15:44:15.597063 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:44:15.597199 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2095,"mutex_wait_us":169,"thread_start_us":1931,"threads_started":1}
I20260928 15:44:15.597894 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.588480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:34285 (local address 127.0.0.1:47264)
0928 15:44:15.590348 (+ 1868us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:15.590367 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:15.590480 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:15.591037 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:15.591042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:15.591076 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:15.591411 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.591421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.593313 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.593324 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.595231 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.595242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.597037 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.597043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.597222 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.597268 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 15:44:15.597372 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:44:15.597493 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1757,"mutex_wait_us":1,"thread_start_us":1591,"threads_started":1}
I20260928 15:44:15.598101 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.585778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.13.5.67:33257)
0928 15:44:15.586978 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.586987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.588161 (+ 1174us) server_negotiation.cc:407] Connection header received
0928 15:44:15.588478 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.588486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.588773 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.588958 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:15.590258 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.591753 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.594385 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.595770 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.595925 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.596169 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.597563 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.597742 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":866,"threads_started":1}
I20260928 15:44:15.598598 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.586249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.13.5.66:43349)
0928 15:44:15.587376 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.587384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.588424 (+ 1040us) server_negotiation.cc:407] Connection header received
0928 15:44:15.588763 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.588772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.589019 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.589196 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:15.590288 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.591600 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.594771 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.596135 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.596264 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.598076 (+ 1812us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.598149 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.598325 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":873,"threads_started":1}
I20260928 15:44:15.598858 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.588813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.13.5.65:34285)
0928 15:44:15.589901 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.589910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.590477 (+ 567us) server_negotiation.cc:407] Connection header received
0928 15:44:15.590604 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.590614 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.590840 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.591025 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:15.591567 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.593161 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.595377 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.596600 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.596710 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.596882 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.598383 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.598558 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":784,"threads_started":1}
I20260928 15:44:15.601989 15496 tablet_service.cc:1467] Processing CreateTablet for tablet ec56579f8e4e46ec90408a35217fec97 (DEFAULT_TABLE table=test-table [id=fec34b1ffa374dd5acd0e76f959bbf77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:15.601987 15357 tablet_service.cc:1467] Processing CreateTablet for tablet ec56579f8e4e46ec90408a35217fec97 (DEFAULT_TABLE table=test-table [id=fec34b1ffa374dd5acd0e76f959bbf77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:15.602190 15218 tablet_service.cc:1467] Processing CreateTablet for tablet ec56579f8e4e46ec90408a35217fec97 (DEFAULT_TABLE table=test-table [id=fec34b1ffa374dd5acd0e76f959bbf77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:15.603333 15218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec56579f8e4e46ec90408a35217fec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:15.603399 15496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec56579f8e4e46ec90408a35217fec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:15.603401 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec56579f8e4e46ec90408a35217fec97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:15.615393 15585 tablet_bootstrap.cc:492] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Bootstrap starting.
I20260928 15:44:15.617690 15586 tablet_bootstrap.cc:492] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Bootstrap starting.
I20260928 15:44:15.618659 15585 tablet_bootstrap.cc:654] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:15.620014 15585 log.cc:824] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:15.621315 15587 tablet_bootstrap.cc:492] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Bootstrap starting.
I20260928 15:44:15.621706 15586 tablet_bootstrap.cc:654] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:15.623106 15586 log.cc:824] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:15.623236 15585 tablet_bootstrap.cc:492] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: No bootstrap required, opened a new log
I20260928 15:44:15.623513 15585 ts_tablet_manager.cc:1397] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:44:15.624812 15587 tablet_bootstrap.cc:654] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:15.626248 15587 log.cc:824] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:15.629091 15587 tablet_bootstrap.cc:492] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: No bootstrap required, opened a new log
I20260928 15:44:15.629374 15587 ts_tablet_manager.cc:1397] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:44:15.629496 15586 tablet_bootstrap.cc:492] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: No bootstrap required, opened a new log
I20260928 15:44:15.629762 15586 ts_tablet_manager.cc:1397] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 15:44:15.633033 15585 raft_consensus.cc:348] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.633472 15585 raft_consensus.cc:374] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:15.633564 15585 raft_consensus.cc:729] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e6e39328174accadf9d1990a77d42f, State: Initialized, Role: FOLLOWER
I20260928 15:44:15.634297 15585 consensus_queue.cc:260] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.636053 15585 ts_tablet_manager.cc:1428] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
I20260928 15:44:15.638182 15587 raft_consensus.cc:348] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.638528 15587 raft_consensus.cc:374] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:15.638648 15587 raft_consensus.cc:729] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c622bb34dcf4baaa83492150ca5044d, State: Initialized, Role: FOLLOWER
I20260928 15:44:15.639298 15587 consensus_queue.cc:260] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.640396 15427 heartbeater.cc:503] Master 127.13.5.126:33683 was elected leader, sending a full tablet report...
I20260928 15:44:15.641000 15587 ts_tablet_manager.cc:1428] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 15:44:15.641681 15586 raft_consensus.cc:348] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.642102 15586 raft_consensus.cc:374] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:15.642199 15586 raft_consensus.cc:729] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff611e9554fb41c381cea809b2e7fbe0, State: Initialized, Role: FOLLOWER
I20260928 15:44:15.642752 15586 consensus_queue.cc:260] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.643781 15566 heartbeater.cc:503] Master 127.13.5.126:33683 was elected leader, sending a full tablet report...
I20260928 15:44:15.644405 15586 ts_tablet_manager.cc:1428] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260928 15:44:15.698587 15289 tablet.cc:2369] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:15.731966 15428 tablet.cc:2369] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:15.756518 15567 tablet.cc:2369] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:44:15.832789 15593 raft_consensus.cc:482] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:15.833178 15593 raft_consensus.cc:504] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.834955 15593 leader_election.cc:288] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349), 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285)
I20260928 15:44:15.842727 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.835319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:46981 (local address 127.13.5.66:43349)
0928 15:44:15.835534 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.835543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.835637 (+ 94us) server_negotiation.cc:407] Connection header received
0928 15:44:15.835935 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.835943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.836025 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.836120 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:15.836765 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.837878 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.840284 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.841618 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.841756 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.841904 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.842235 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.842462 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:44:15.842725 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.836039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:52107 (local address 127.13.5.65:34285)
0928 15:44:15.836304 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.836309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.836340 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:44:15.836467 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.836470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.836544 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.836632 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:15.837138 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.838116 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.840468 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.841751 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.841880 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.841985 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.842329 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.842462 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 15:44:15.842784 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.835278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43349 (local address 127.13.5.67:46981)
0928 15:44:15.835512 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:15.835544 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:15.835711 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:15.836294 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:15.836301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:15.836329 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:15.836571 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.836580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.838021 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.838025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.840116 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.840136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.841802 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.841807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.842047 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.842087 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:44:15.842188 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:44:15.842344 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:44:15.842868 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.835423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:34285 (local address 127.13.5.67:52107)
0928 15:44:15.836247 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:15.836264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:15.836348 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:15.836681 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:15.836685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:15.836711 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:15.836987 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.836993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.838240 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.838245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.840228 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.840258 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.841957 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.841963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.842113 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.842155 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:44:15.842274 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 15:44:15.842374 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":464,"threads_started":1}
I20260928 15:44:15.844010 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec56579f8e4e46ec90408a35217fec97" candidate_uuid: "ff611e9554fb41c381cea809b2e7fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c622bb34dcf4baaa83492150ca5044d" is_pre_election: true
I20260928 15:44:15.844110 15242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec56579f8e4e46ec90408a35217fec97" candidate_uuid: "ff611e9554fb41c381cea809b2e7fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f" is_pre_election: true
I20260928 15:44:15.844551 15382 raft_consensus.cc:2454] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff611e9554fb41c381cea809b2e7fbe0 in term 0.
I20260928 15:44:15.844600 15242 raft_consensus.cc:2454] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff611e9554fb41c381cea809b2e7fbe0 in term 0.
I20260928 15:44:15.845216 15447 leader_election.cc:302] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c622bb34dcf4baaa83492150ca5044d, ff611e9554fb41c381cea809b2e7fbe0; no voters:
I20260928 15:44:15.845690 15593 raft_consensus.cc:2781] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:15.845865 15593 raft_consensus.cc:482] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:15.845973 15593 raft_consensus.cc:3037] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:15.848379 15593 raft_consensus.cc:504] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.849028 15593 leader_election.cc:288] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349), 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285)
I20260928 15:44:15.849534 15242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec56579f8e4e46ec90408a35217fec97" candidate_uuid: "ff611e9554fb41c381cea809b2e7fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f"
I20260928 15:44:15.849776 15242 raft_consensus.cc:3037] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:15.849813 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec56579f8e4e46ec90408a35217fec97" candidate_uuid: "ff611e9554fb41c381cea809b2e7fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c622bb34dcf4baaa83492150ca5044d"
I20260928 15:44:15.850040 15382 raft_consensus.cc:3037] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:15.853186 15242 raft_consensus.cc:2454] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff611e9554fb41c381cea809b2e7fbe0 in term 1.
I20260928 15:44:15.853186 15382 raft_consensus.cc:2454] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff611e9554fb41c381cea809b2e7fbe0 in term 1.
I20260928 15:44:15.853749 15449 leader_election.cc:302] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e6e39328174accadf9d1990a77d42f, ff611e9554fb41c381cea809b2e7fbe0; no voters:
I20260928 15:44:15.854063 15593 raft_consensus.cc:2781] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:15.854537 15593 raft_consensus.cc:686] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: ff611e9554fb41c381cea809b2e7fbe0, State: Running, Role: LEADER
I20260928 15:44:15.855037 15593 consensus_queue.cc:237] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } }
I20260928 15:44:15.860226 15081 catalog_manager.cc:5495] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "ff611e9554fb41c381cea809b2e7fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:44:15 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610253, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:15.933698 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.918603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33683 (local address 127.0.0.1:44410)
0928 15:44:15.919376 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:15.919392 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:15.919527 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:15.919847 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:15.919850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:15.920608 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:15.920956 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.920967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.922255 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.922296 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.923282 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:15.923294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.924298 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.924305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:15.924435 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.925097 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:15.925114 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:15.927454 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:15.930335 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:15.930347 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:15.930353 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:15.930775 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:15.931341 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:15.931345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:15.931348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:15.931564 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:15.932330 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:15.932339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:15.932658 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 15:44:15.932896 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 15:44:15.933229 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":456,"threads_started":1}
I20260928 15:44:15.933946 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:15.918516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.13.5.126:33683)
0928 15:44:15.918766 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:15.918774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:15.919486 (+ 712us) server_negotiation.cc:407] Connection header received
0928 15:44:15.919634 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:15.919638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:15.919699 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:15.919782 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:15.921205 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.922130 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.923504 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:15.924071 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:15.924225 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:15.927654 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:15.927659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:15.927662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:15.927691 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:15.930193 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:15.930904 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:15.930908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:15.930910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:15.931036 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:15.931741 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:15.931748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:15.931752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:15.932181 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:15.932363 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:15.933003 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 15:44:15.933524 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":298,"server-negotiator.queue_time_us":88}
I20260928 15:44:15.939996 15081 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:44410:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:44:15.941051 15081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:15.947773 15357 tablet_service.cc:1467] Processing CreateTablet for tablet 06a2ea20fe884a2882c6100904b1ff6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a66468663d9b455e99acdb47da1cd5b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:44:15.947685 15218 tablet_service.cc:1467] Processing CreateTablet for tablet 06a2ea20fe884a2882c6100904b1ff6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a66468663d9b455e99acdb47da1cd5b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:44:15.948047 15496 tablet_service.cc:1467] Processing CreateTablet for tablet 06a2ea20fe884a2882c6100904b1ff6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a66468663d9b455e99acdb47da1cd5b5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:44:15.948801 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a2ea20fe884a2882c6100904b1ff6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:15.948814 15218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a2ea20fe884a2882c6100904b1ff6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:15.948912 15496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a2ea20fe884a2882c6100904b1ff6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:15.956163 15585 tablet_bootstrap.cc:492] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: Bootstrap starting.
I20260928 15:44:15.956166 15586 tablet_bootstrap.cc:492] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: Bootstrap starting.
I20260928 15:44:15.957860 15585 tablet_bootstrap.cc:654] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:15.957860 15586 tablet_bootstrap.cc:654] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:15.959038 15587 tablet_bootstrap.cc:492] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: Bootstrap starting.
I20260928 15:44:15.960984 15587 tablet_bootstrap.cc:654] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:15.960989 15585 tablet_bootstrap.cc:492] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: No bootstrap required, opened a new log
I20260928 15:44:15.961141 15585 ts_tablet_manager.cc:1397] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 15:44:15.961337 15586 tablet_bootstrap.cc:492] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: No bootstrap required, opened a new log
I20260928 15:44:15.961517 15586 ts_tablet_manager.cc:1397] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 15:44:15.962060 15585 raft_consensus.cc:348] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:15.962387 15585 raft_consensus.cc:374] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:15.962491 15585 raft_consensus.cc:729] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e6e39328174accadf9d1990a77d42f, State: Initialized, Role: FOLLOWER
I20260928 15:44:15.962371 15586 raft_consensus.cc:348] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:15.962635 15586 raft_consensus.cc:374] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:15.962741 15586 raft_consensus.cc:729] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff611e9554fb41c381cea809b2e7fbe0, State: Initialized, Role: FOLLOWER
I20260928 15:44:15.962857 15585 consensus_queue.cc:260] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:15.962983 15586 consensus_queue.cc:260] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:15.963379 15593 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:15.963449 15591 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:15.963745 15593 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:15.963886 15591 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:15.964854 15586 ts_tablet_manager.cc:1428] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:15.967355 15585 ts_tablet_manager.cc:1428] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260928 15:44:15.968514 15587 tablet_bootstrap.cc:492] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: No bootstrap required, opened a new log
I20260928 15:44:15.968705 15587 ts_tablet_manager.cc:1397] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 15:44:15.969498 15587 raft_consensus.cc:348] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:15.969761 15587 raft_consensus.cc:374] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:15.969869 15587 raft_consensus.cc:729] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c622bb34dcf4baaa83492150ca5044d, State: Initialized, Role: FOLLOWER
I20260928 15:44:15.970109 15587 consensus_queue.cc:260] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:15.970768 15587 ts_tablet_manager.cc:1428] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 15:44:15.970642 15592 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:15.971096 15592 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:16.010757 15592 raft_consensus.cc:482] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:16.011031 15592 raft_consensus.cc:504] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.012636 15592 leader_election.cc:288] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285), ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257)
I20260928 15:44:16.019573 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.013018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:48411 (local address 127.13.5.65:34285)
0928 15:44:16.013258 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:16.013267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:16.013605 (+ 338us) server_negotiation.cc:407] Connection header received
0928 15:44:16.013861 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:16.013869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:16.013973 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:16.014101 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:16.014774 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.015753 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.017689 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.018545 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.018644 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.018721 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:16.019131 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 15:44:16.019271 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:44:16.019626 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.012967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:34285 (local address 127.13.5.66:48411)
0928 15:44:16.013494 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:16.013517 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:16.013677 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:16.014146 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:16.014150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:16.014182 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:16.014602 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.014616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.015912 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.015918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.017526 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.017539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.018763 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.018770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.018944 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.018981 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:44:16.019082 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:44:16.019215 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":4,"thread_start_us":284,"threads_started":1}
I20260928 15:44:16.020198 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.013701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:33257 (local address 127.13.5.66:55875)
0928 15:44:16.014320 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:16.014336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:16.014431 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:16.014766 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:16.014770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:16.014794 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:16.015157 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.015169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.016299 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.016303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.017740 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.017753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.019025 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.019030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.019275 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.019307 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:44:16.019407 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:44:16.019580 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":41,"thread_start_us":289,"threads_started":1}
I20260928 15:44:16.020452 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.013819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:55875 (local address 127.13.5.67:33257)
0928 15:44:16.013995 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:16.013999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:16.014428 (+ 429us) server_negotiation.cc:407] Connection header received
0928 15:44:16.014544 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:16.014555 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:16.014633 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:16.014732 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:16.015327 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.016177 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.017911 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.018807 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.018941 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.019049 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:16.019453 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 15:44:16.019593 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:44:16.020434 15242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a2ea20fe884a2882c6100904b1ff6e" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f" is_pre_election: true
I20260928 15:44:16.020723 15242 raft_consensus.cc:2454] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 0.
I20260928 15:44:16.021268 15310 leader_election.cc:302] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c622bb34dcf4baaa83492150ca5044d, 67e6e39328174accadf9d1990a77d42f; no voters:
I20260928 15:44:16.021302 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a2ea20fe884a2882c6100904b1ff6e" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff611e9554fb41c381cea809b2e7fbe0" is_pre_election: true
I20260928 15:44:16.021724 15516 raft_consensus.cc:2454] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 0.
I20260928 15:44:16.021776 15592 raft_consensus.cc:2781] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:16.021891 15592 raft_consensus.cc:482] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:16.021996 15592 raft_consensus.cc:3037] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:16.024333 15592 raft_consensus.cc:504] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.024870 15592 leader_election.cc:288] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 election: Requested vote from peers 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285), ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257)
I20260928 15:44:16.025460 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a2ea20fe884a2882c6100904b1ff6e" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff611e9554fb41c381cea809b2e7fbe0"
I20260928 15:44:16.025652 15516 raft_consensus.cc:3037] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:16.025574 15242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a2ea20fe884a2882c6100904b1ff6e" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f"
I20260928 15:44:16.025853 15242 raft_consensus.cc:3037] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:16.027698 15516 raft_consensus.cc:2454] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 1.
I20260928 15:44:16.028157 15242 raft_consensus.cc:2454] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 1.
I20260928 15:44:16.028182 15310 leader_election.cc:302] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c622bb34dcf4baaa83492150ca5044d, ff611e9554fb41c381cea809b2e7fbe0; no voters:
I20260928 15:44:16.028525 15592 raft_consensus.cc:2781] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:16.029042 15592 raft_consensus.cc:686] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [term 1 LEADER]: Becoming Leader. State: Replica: 0c622bb34dcf4baaa83492150ca5044d, State: Running, Role: LEADER
I20260928 15:44:16.029529 15592 consensus_queue.cc:237] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.032752 15608 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: TxnStatusTablet state changed. Reason: New leader 0c622bb34dcf4baaa83492150ca5044d. Latest consensus state: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:16.033102 15608 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:44:16.033908 15610 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:44:16.034168 15610 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:44:16.034683 15081 catalog_manager.cc:5495] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66). New cstate: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } health_report { overall_health: UNKNOWN } } }
I20260928 15:44:16.071823 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.066204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43349 (local address 127.0.0.1:56020)
0928 15:44:16.066433 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:16.066450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:16.066634 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:16.067096 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:16.067101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:16.067112 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:44:16.067308 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.067317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.068385 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.068392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.069417 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.069428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.070137 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.070144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.070320 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.070375 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:44:16.071260 (+ 885us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:44:16.071284 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:44:16.071370 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:44:16.071525 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 15:44:16.071839 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.066133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.13.5.66:43349)
0928 15:44:16.066329 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:16.066337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:16.066556 (+ 219us) server_negotiation.cc:407] Connection header received
0928 15:44:16.066850 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:16.066854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:16.066929 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:16.067082 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:16.067439 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.068259 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.069564 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.069979 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.070111 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.070578 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:16.071150 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:16.071309 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:16.071387 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:44:16.071477 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:44:16.080592 15516 raft_consensus.cc:1261] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 0c622bb34dcf4baaa83492150ca5044d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:16.080740 15242 raft_consensus.cc:1261] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Refusing update from remote peer 0c622bb34dcf4baaa83492150ca5044d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:16.081580 15608 consensus_queue.cc:1035] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:16.082000 15592 consensus_queue.cc:1035] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:16.093483 15593 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: TxnStatusTablet state changed. Reason: New leader 0c622bb34dcf4baaa83492150ca5044d. Latest consensus state: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:16.093807 15593 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:16.103492 15592 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:16.103814 15592 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:44:16.106920 15591 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: TxnStatusTablet state changed. Reason: New leader 0c622bb34dcf4baaa83492150ca5044d. Latest consensus state: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:16.107231 15591 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:16.108495 15593 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:16.108837 15593 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P ff611e9554fb41c381cea809b2e7fbe0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:16.112922 15615 mvcc.cc:205] Tried to move back new op lower bound from 7334339608896430080 to 7334339608700502016. Current Snapshot: MvccSnapshot[applied={T|T < 7334339608896430080}]
I20260928 15:44:16.113227 15608 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:16.113482 15608 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P 0c622bb34dcf4baaa83492150ca5044d: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:44:16.115504 15591 tablet_replica.cc:439] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } } }
I20260928 15:44:16.115742 15591 tablet_replica.cc:442] T 06a2ea20fe884a2882c6100904b1ff6e P 67e6e39328174accadf9d1990a77d42f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:44:16.126467 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.119144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.13.5.67:33257)
0928 15:44:16.119339 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:16.119348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:16.120152 (+ 804us) server_negotiation.cc:407] Connection header received
0928 15:44:16.120419 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:16.120423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:16.120525 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:16.120731 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:16.121229 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.122196 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.124255 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.124705 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.124801 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.125285 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:16.125820 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:16.125934 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:16.126075 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:44:16.126144 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:44:16.146165 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.140101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:34713 (local address 127.13.5.66:43349)
0928 15:44:16.140321 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:16.140328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:16.140620 (+ 292us) server_negotiation.cc:407] Connection header received
0928 15:44:16.140900 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:16.140905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:16.141057 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:16.141146 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:16.141849 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.142849 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.144330 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.144764 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.144906 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.145370 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:16.145546 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:16.145640 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:16.145781 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:44:16.145857 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:44:16.146327 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.139500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43349 (local address 127.13.5.67:34713)
0928 15:44:16.140467 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:16.140487 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:16.140698 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:16.141191 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:16.141198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:16.141212 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:44:16.141608 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.141621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.142978 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.142986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.144193 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.144206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.144944 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.144953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.145097 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.145154 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:44:16.145640 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:44:16.145654 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:44:16.145759 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:44:16.145897 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":800,"threads_started":1}
I20260928 15:44:16.156800 15242 raft_consensus.cc:1261] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Refusing update from remote peer ff611e9554fb41c381cea809b2e7fbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:16.156955 15382 raft_consensus.cc:1261] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Refusing update from remote peer ff611e9554fb41c381cea809b2e7fbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:16.157433 15595 consensus_queue.cc:1035] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:16.158377 15595 consensus_queue.cc:1035] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:16.164727 15614 mvcc.cc:205] Tried to move back new op lower bound from 7334339609211703296 to 7334339607985975296. Current Snapshot: MvccSnapshot[applied={T|T < 7334339609211703296}]
I20260928 15:44:16.211769 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.200802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:52541 (local address 127.13.5.67:33257)
0928 15:44:16.201049 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:16.201059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:16.205683 (+ 4624us) server_negotiation.cc:407] Connection header received
0928 15:44:16.206012 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:16.206019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:16.206125 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:16.206229 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:16.206898 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.208128 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.209581 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.210270 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.210424 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.210954 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:16.211100 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:16.211194 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:16.211373 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 15:44:16.211455 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:44:16.212175 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:16.200816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:33257 (local address 127.13.5.66:52541)
0928 15:44:16.205586 (+ 4770us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:16.205613 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:16.205763 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:16.206258 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:16.206262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:16.206271 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:44:16.206731 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.206746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.208252 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.208256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.209440 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:16.209451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:16.210497 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:16.210505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:16.210660 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:16.210758 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:44:16.211216 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:44:16.211230 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:44:16.211346 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:44:16.211482 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4562,"thread_start_us":4387,"threads_started":1}
I20260928 15:44:16.249523 15080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:16.249879 15080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:16.254124 15080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: Sending DeleteTablet for 3 replicas of tablet ec56579f8e4e46ec90408a35217fec97
I20260928 15:44:16.255264 15496 tablet_service.cc:1514] Processing DeleteTablet for tablet ec56579f8e4e46ec90408a35217fec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:16 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47194
I20260928 15:44:16.255578 15218 tablet_service.cc:1514] Processing DeleteTablet for tablet ec56579f8e4e46ec90408a35217fec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:16 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47264
I20260928 15:44:16.256582 15636 tablet_replica.cc:330] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: stopping tablet replica
I20260928 15:44:16.257026 15636 raft_consensus.cc:2229] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:16.257373 15636 raft_consensus.cc:2258] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:16.257843 15080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:16.258692 15080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:16.260740 15357 tablet_service.cc:1514] Processing DeleteTablet for tablet ec56579f8e4e46ec90408a35217fec97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:16 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:56012
I20260928 15:44:16.267527 15636 ts_tablet_manager.cc:1905] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:16.268913 15218 tablet_service.cc:1467] Processing CreateTablet for tablet 60c3db874585436b84a6c2fd5c3b6136 (DEFAULT_TABLE table=test-table [id=920799c6d2a647608136c8148f19f0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:16.269393 15218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c3db874585436b84a6c2fd5c3b6136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:16.274426 15637 tablet_replica.cc:330] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: stopping tablet replica
I20260928 15:44:16.274924 15637 raft_consensus.cc:2229] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:16.275355 15637 raft_consensus.cc:2258] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:16.276927 15357 tablet_service.cc:1467] Processing CreateTablet for tablet 60c3db874585436b84a6c2fd5c3b6136 (DEFAULT_TABLE table=test-table [id=920799c6d2a647608136c8148f19f0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:16.277393 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c3db874585436b84a6c2fd5c3b6136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:16.279577 15495 tablet_service.cc:1467] Processing CreateTablet for tablet 60c3db874585436b84a6c2fd5c3b6136 (DEFAULT_TABLE table=test-table [id=920799c6d2a647608136c8148f19f0af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:16.280193 15495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c3db874585436b84a6c2fd5c3b6136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:16.282176 15585 tablet_bootstrap.cc:492] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: Bootstrap starting.
I20260928 15:44:16.283913 15585 tablet_bootstrap.cc:654] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:16.284600 15587 tablet_bootstrap.cc:492] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: Bootstrap starting.
I20260928 15:44:16.286327 15587 tablet_bootstrap.cc:654] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:16.287370 15586 tablet_bootstrap.cc:492] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: Bootstrap starting.
I20260928 15:44:16.289342 15586 tablet_bootstrap.cc:654] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:16.293187 15636 ts_tablet_manager.cc:1918] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:16.294107 15636 log.cc:1196] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/ec56579f8e4e46ec90408a35217fec97
I20260928 15:44:16.294870 15636 ts_tablet_manager.cc:1939] T ec56579f8e4e46ec90408a35217fec97 P 67e6e39328174accadf9d1990a77d42f: Deleting consensus metadata
I20260928 15:44:16.296361 15635 tablet_replica.cc:330] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: stopping tablet replica
I20260928 15:44:16.296936 15635 raft_consensus.cc:2229] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:44:16.297734 15635 raft_consensus.cc:2258] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:16.299849 15067 catalog_manager.cc:4841] TS 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285): tablet ec56579f8e4e46ec90408a35217fec97 (table test-table [id=fec34b1ffa374dd5acd0e76f959bbf77]) successfully deleted
I20260928 15:44:16.300097 15637 ts_tablet_manager.cc:1905] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:16.300511 15587 tablet_bootstrap.cc:492] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: No bootstrap required, opened a new log
I20260928 15:44:16.300740 15587 ts_tablet_manager.cc:1397] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260928 15:44:16.301179 15585 tablet_bootstrap.cc:492] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: No bootstrap required, opened a new log
I20260928 15:44:16.301366 15585 ts_tablet_manager.cc:1397] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20260928 15:44:16.301826 15587 raft_consensus.cc:348] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.302179 15587 raft_consensus.cc:374] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:16.302116 15585 raft_consensus.cc:348] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.302361 15585 raft_consensus.cc:374] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:16.302376 15587 raft_consensus.cc:729] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c622bb34dcf4baaa83492150ca5044d, State: Initialized, Role: FOLLOWER
I20260928 15:44:16.302441 15585 raft_consensus.cc:729] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e6e39328174accadf9d1990a77d42f, State: Initialized, Role: FOLLOWER
I20260928 15:44:16.302752 15585 consensus_queue.cc:260] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.302788 15587 consensus_queue.cc:260] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.304298 15587 ts_tablet_manager.cc:1428] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:44:16.305533 15585 ts_tablet_manager.cc:1428] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 15:44:16.307547 15635 ts_tablet_manager.cc:1905] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:16.310385 15637 ts_tablet_manager.cc:1918] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:16.310632 15637 log.cc:1196] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/ec56579f8e4e46ec90408a35217fec97
I20260928 15:44:16.311255 15637 ts_tablet_manager.cc:1939] T ec56579f8e4e46ec90408a35217fec97 P 0c622bb34dcf4baaa83492150ca5044d: Deleting consensus metadata
I20260928 15:44:16.313128 15065 catalog_manager.cc:4841] TS 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349): tablet ec56579f8e4e46ec90408a35217fec97 (table test-table [id=fec34b1ffa374dd5acd0e76f959bbf77]) successfully deleted
I20260928 15:44:16.313834 15586 tablet_bootstrap.cc:492] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: No bootstrap required, opened a new log
I20260928 15:44:16.314044 15586 ts_tablet_manager.cc:1397] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20260928 15:44:16.314723 15586 raft_consensus.cc:348] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.314952 15586 raft_consensus.cc:374] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:16.315062 15586 raft_consensus.cc:729] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff611e9554fb41c381cea809b2e7fbe0, State: Initialized, Role: FOLLOWER
I20260928 15:44:16.315328 15586 consensus_queue.cc:260] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.316088 15586 ts_tablet_manager.cc:1428] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:44:16.317727 15635 ts_tablet_manager.cc:1918] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:16.317899 15635 log.cc:1196] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/ec56579f8e4e46ec90408a35217fec97
I20260928 15:44:16.318481 15635 ts_tablet_manager.cc:1939] T ec56579f8e4e46ec90408a35217fec97 P ff611e9554fb41c381cea809b2e7fbe0: Deleting consensus metadata
I20260928 15:44:16.318955 15593 raft_consensus.cc:482] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:16.319115 15593 raft_consensus.cc:504] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.319803 15593 leader_election.cc:288] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285), 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349)
I20260928 15:44:16.320657 15242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c3db874585436b84a6c2fd5c3b6136" candidate_uuid: "ff611e9554fb41c381cea809b2e7fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f" is_pre_election: true
I20260928 15:44:16.320799 15067 catalog_manager.cc:4841] TS ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257): tablet ec56579f8e4e46ec90408a35217fec97 (table test-table [id=fec34b1ffa374dd5acd0e76f959bbf77]) successfully deleted
I20260928 15:44:16.320917 15242 raft_consensus.cc:2454] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff611e9554fb41c381cea809b2e7fbe0 in term 0.
I20260928 15:44:16.321406 15449 leader_election.cc:302] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e6e39328174accadf9d1990a77d42f, ff611e9554fb41c381cea809b2e7fbe0; no voters:
I20260928 15:44:16.322022 15593 raft_consensus.cc:2781] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:16.322146 15593 raft_consensus.cc:482] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:16.322271 15593 raft_consensus.cc:3037] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:16.324307 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c3db874585436b84a6c2fd5c3b6136" candidate_uuid: "ff611e9554fb41c381cea809b2e7fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c622bb34dcf4baaa83492150ca5044d" is_pre_election: true
I20260928 15:44:16.324630 15382 raft_consensus.cc:2454] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff611e9554fb41c381cea809b2e7fbe0 in term 0.
I20260928 15:44:16.324725 15593 raft_consensus.cc:504] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.326225 15242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c3db874585436b84a6c2fd5c3b6136" candidate_uuid: "ff611e9554fb41c381cea809b2e7fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f"
I20260928 15:44:16.326459 15242 raft_consensus.cc:3037] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:16.328562 15242 raft_consensus.cc:2454] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff611e9554fb41c381cea809b2e7fbe0 in term 1.
I20260928 15:44:16.329115 15449 leader_election.cc:302] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e6e39328174accadf9d1990a77d42f, ff611e9554fb41c381cea809b2e7fbe0; no voters:
I20260928 15:44:16.329409 15595 raft_consensus.cc:2781] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:16.329605 15595 raft_consensus.cc:686] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 LEADER]: Becoming Leader. State: Replica: ff611e9554fb41c381cea809b2e7fbe0, State: Running, Role: LEADER
I20260928 15:44:16.329895 15595 consensus_queue.cc:237] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:16.331936 15593 leader_election.cc:288] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [CANDIDATE]: Term 1 election: Requested vote from peers 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285), 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349)
I20260928 15:44:16.332064 15382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c3db874585436b84a6c2fd5c3b6136" candidate_uuid: "ff611e9554fb41c381cea809b2e7fbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c622bb34dcf4baaa83492150ca5044d"
I20260928 15:44:16.332348 15382 raft_consensus.cc:3037] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:16.336741 15382 raft_consensus.cc:2454] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff611e9554fb41c381cea809b2e7fbe0 in term 1.
I20260928 15:44:16.341948 15080 catalog_manager.cc:5495] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "ff611e9554fb41c381cea809b2e7fbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } health_report { overall_health: HEALTHY } } }
I20260928 15:44:16.379303 15242 raft_consensus.cc:1261] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Refusing update from remote peer ff611e9554fb41c381cea809b2e7fbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:16.380494 15595 consensus_queue.cc:1035] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:16.385035 15382 raft_consensus.cc:1261] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Refusing update from remote peer ff611e9554fb41c381cea809b2e7fbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:16.385947 15595 consensus_queue.cc:1035] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:16.409361 15613 mvcc.cc:205] Tried to move back new op lower bound from 7334339610123358208 to 7334339609930170368. Current Snapshot: MvccSnapshot[applied={T|T < 7334339610123358208}]
W20260928 15:44:16.415958 15614 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 15:44:18.038308 15080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:18.038606 15080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:18.043102 15080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: Sending DeleteTablet for 3 replicas of tablet 60c3db874585436b84a6c2fd5c3b6136
I20260928 15:44:18.044067 15496 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c3db874585436b84a6c2fd5c3b6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:18 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47194
I20260928 15:44:18.044061 15218 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c3db874585436b84a6c2fd5c3b6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:18 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47264
I20260928 15:44:18.044137 15357 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c3db874585436b84a6c2fd5c3b6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:18 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:56012
I20260928 15:44:18.057895 15667 tablet_replica.cc:330] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: stopping tablet replica
I20260928 15:44:18.058379 15667 raft_consensus.cc:2229] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:44:18.058948 15667 raft_consensus.cc:2258] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:18.059389 15668 tablet_replica.cc:330] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: stopping tablet replica
I20260928 15:44:18.059590 15668 raft_consensus.cc:2229] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:18.059964 15668 raft_consensus.cc:2258] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:18.060155 15667 ts_tablet_manager.cc:1905] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:18.061391 15668 ts_tablet_manager.cc:1905] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:18.064693 15666 tablet_replica.cc:330] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: stopping tablet replica
I20260928 15:44:18.064975 15666 raft_consensus.cc:2229] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:18.065296 15666 raft_consensus.cc:2258] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:18.066542 15666 ts_tablet_manager.cc:1905] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:18.068291 15667 ts_tablet_manager.cc:1918] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:44:18.068427 15668 ts_tablet_manager.cc:1918] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:44:18.068475 15667 log.cc:1196] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/60c3db874585436b84a6c2fd5c3b6136
I20260928 15:44:18.068634 15668 log.cc:1196] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/60c3db874585436b84a6c2fd5c3b6136
I20260928 15:44:18.068989 15667 ts_tablet_manager.cc:1939] T 60c3db874585436b84a6c2fd5c3b6136 P ff611e9554fb41c381cea809b2e7fbe0: Deleting consensus metadata
I20260928 15:44:18.069169 15668 ts_tablet_manager.cc:1939] T 60c3db874585436b84a6c2fd5c3b6136 P 0c622bb34dcf4baaa83492150ca5044d: Deleting consensus metadata
I20260928 15:44:18.070286 15065 catalog_manager.cc:4841] TS 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349): tablet 60c3db874585436b84a6c2fd5c3b6136 (table test-table [id=920799c6d2a647608136c8148f19f0af]) successfully deleted
I20260928 15:44:18.070282 15067 catalog_manager.cc:4841] TS ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257): tablet 60c3db874585436b84a6c2fd5c3b6136 (table test-table [id=920799c6d2a647608136c8148f19f0af]) successfully deleted
I20260928 15:44:18.072647 15666 ts_tablet_manager.cc:1918] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:44:18.072809 15666 log.cc:1196] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/60c3db874585436b84a6c2fd5c3b6136
I20260928 15:44:18.073189 15666 ts_tablet_manager.cc:1939] T 60c3db874585436b84a6c2fd5c3b6136 P 67e6e39328174accadf9d1990a77d42f: Deleting consensus metadata
I20260928 15:44:18.074221 15067 catalog_manager.cc:4841] TS 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285): tablet 60c3db874585436b84a6c2fd5c3b6136 (table test-table [id=920799c6d2a647608136c8148f19f0af]) successfully deleted
Sep 28 15:44:18 dist-test-slave-j968 krb5kdc[15036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:44:18 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:18.105865 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:18.090879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.13.5.126:33683)
0928 15:44:18.091458 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:18.091465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:18.091585 (+ 120us) server_negotiation.cc:407] Connection header received
0928 15:44:18.091816 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:18.091820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:18.091946 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:18.092068 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:18.093502 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:18.094327 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:18.095593 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:18.096119 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:18.096255 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:18.100300 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:18.100305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:18.100309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:18.100341 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:18.103305 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:18.104015 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:18.104020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:18.104022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:18.104148 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:18.104530 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:18.104533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:18.104535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:18.104815 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:18.105000 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:18.105321 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 15:44:18.105493 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":276,"threads_started":1}
I20260928 15:44:18.109694 15080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:18.110383 15080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:18.116899 15218 tablet_service.cc:1467] Processing CreateTablet for tablet 53da65e5ed224675821166d313490f6d (DEFAULT_TABLE table=test-table [id=d48dc7dac7c749599c7b488773823cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:18.116983 15357 tablet_service.cc:1467] Processing CreateTablet for tablet 53da65e5ed224675821166d313490f6d (DEFAULT_TABLE table=test-table [id=d48dc7dac7c749599c7b488773823cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:18.117133 15496 tablet_service.cc:1467] Processing CreateTablet for tablet 53da65e5ed224675821166d313490f6d (DEFAULT_TABLE table=test-table [id=d48dc7dac7c749599c7b488773823cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:18.117421 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53da65e5ed224675821166d313490f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:18.117441 15218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53da65e5ed224675821166d313490f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:18.117522 15496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53da65e5ed224675821166d313490f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:18.124112 15677 tablet_bootstrap.cc:492] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: Bootstrap starting.
I20260928 15:44:18.124799 15678 tablet_bootstrap.cc:492] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: Bootstrap starting.
I20260928 15:44:18.125553 15679 tablet_bootstrap.cc:492] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: Bootstrap starting.
I20260928 15:44:18.126130 15677 tablet_bootstrap.cc:654] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:18.126624 15678 tablet_bootstrap.cc:654] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:18.127475 15679 tablet_bootstrap.cc:654] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:18.128501 15677 tablet_bootstrap.cc:492] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: No bootstrap required, opened a new log
I20260928 15:44:18.128692 15677 ts_tablet_manager.cc:1397] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 15:44:18.129760 15678 tablet_bootstrap.cc:492] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: No bootstrap required, opened a new log
I20260928 15:44:18.129706 15677 raft_consensus.cc:348] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.129922 15679 tablet_bootstrap.cc:492] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: No bootstrap required, opened a new log
I20260928 15:44:18.129940 15678 ts_tablet_manager.cc:1397] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 15:44:18.129999 15677 raft_consensus.cc:374] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:18.130085 15679 ts_tablet_manager.cc:1397] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 15:44:18.130134 15677 raft_consensus.cc:729] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e6e39328174accadf9d1990a77d42f, State: Initialized, Role: FOLLOWER
I20260928 15:44:18.130429 15677 consensus_queue.cc:260] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.130945 15678 raft_consensus.cc:348] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.131290 15678 raft_consensus.cc:374] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:18.131102 15679 raft_consensus.cc:348] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.131439 15679 raft_consensus.cc:374] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:18.131537 15678 raft_consensus.cc:729] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c622bb34dcf4baaa83492150ca5044d, State: Initialized, Role: FOLLOWER
I20260928 15:44:18.131932 15677 ts_tablet_manager.cc:1428] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:18.132289 15678 consensus_queue.cc:260] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.133355 15678 ts_tablet_manager.cc:1428] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:44:18.131630 15679 raft_consensus.cc:729] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff611e9554fb41c381cea809b2e7fbe0, State: Initialized, Role: FOLLOWER
I20260928 15:44:18.136320 15679 consensus_queue.cc:260] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.136978 15679 ts_tablet_manager.cc:1428] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260928 15:44:18.204172 15647 raft_consensus.cc:482] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:18.204447 15647 raft_consensus.cc:504] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.205240 15647 leader_election.cc:288] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285), ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257)
I20260928 15:44:18.205804 15242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53da65e5ed224675821166d313490f6d" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f" is_pre_election: true
I20260928 15:44:18.205858 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53da65e5ed224675821166d313490f6d" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff611e9554fb41c381cea809b2e7fbe0" is_pre_election: true
I20260928 15:44:18.206043 15242 raft_consensus.cc:2454] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 0.
I20260928 15:44:18.206185 15516 raft_consensus.cc:2454] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 0.
I20260928 15:44:18.206708 15310 leader_election.cc:302] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c622bb34dcf4baaa83492150ca5044d, 67e6e39328174accadf9d1990a77d42f; no voters:
I20260928 15:44:18.206982 15647 raft_consensus.cc:2781] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:18.207111 15647 raft_consensus.cc:482] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:18.207209 15647 raft_consensus.cc:3037] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:18.209475 15647 raft_consensus.cc:504] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.210109 15647 leader_election.cc:288] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 election: Requested vote from peers 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285), ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257)
I20260928 15:44:18.210644 15242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53da65e5ed224675821166d313490f6d" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f"
I20260928 15:44:18.210747 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53da65e5ed224675821166d313490f6d" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff611e9554fb41c381cea809b2e7fbe0"
I20260928 15:44:18.210882 15242 raft_consensus.cc:3037] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:18.211014 15516 raft_consensus.cc:3037] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:18.213104 15516 raft_consensus.cc:2454] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 1.
I20260928 15:44:18.213111 15242 raft_consensus.cc:2454] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 1.
I20260928 15:44:18.213521 15310 leader_election.cc:302] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c622bb34dcf4baaa83492150ca5044d, 67e6e39328174accadf9d1990a77d42f; no voters:
I20260928 15:44:18.213856 15647 raft_consensus.cc:2781] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:18.213980 15647 raft_consensus.cc:686] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 1 LEADER]: Becoming Leader. State: Replica: 0c622bb34dcf4baaa83492150ca5044d, State: Running, Role: LEADER
I20260928 15:44:18.214270 15647 consensus_queue.cc:237] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.217080 15080 catalog_manager.cc:5495] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66). New cstate: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } health_report { overall_health: UNKNOWN } } }
I20260928 15:44:18.238646 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:18.233081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.13.5.66:43349)
0928 15:44:18.233861 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:18.233866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:18.233886 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:44:18.233938 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:18.233941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:18.234001 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:18.234084 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:18.234457 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:18.235255 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:18.236703 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:18.237305 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:18.237463 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:18.237829 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:18.237960 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:18.238039 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:18.238168 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 15:44:18.238231 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":517,"threads_started":1}
I20260928 15:44:18.253006 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:18.247605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51065 (local address 127.13.5.66:43349)
0928 15:44:18.247840 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:18.247848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:18.248307 (+ 459us) server_negotiation.cc:407] Connection header received
0928 15:44:18.248509 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:18.248514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:18.248604 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:18.248695 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:18.249168 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:18.249949 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:18.251240 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:18.251612 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:18.251736 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:18.252194 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:18.252289 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:18.252380 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:18.252490 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 15:44:18.252616 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":80}
I20260928 15:44:18.253023 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:18.247574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43349 (local address 127.13.5.66:51065)
0928 15:44:18.248202 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:18.248221 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:18.248382 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:18.248704 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:18.248708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:18.248718 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:44:18.249006 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:18.249019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:18.250079 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:18.250087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:18.251094 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:18.251105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:18.251744 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:18.251749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:18.252001 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:18.252037 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:44:18.252388 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:44:18.252399 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:44:18.252477 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:44:18.252663 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":74,"thread_start_us":405,"threads_started":1}
I20260928 15:44:18.261046 15516 raft_consensus.cc:1261] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 0c622bb34dcf4baaa83492150ca5044d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:18.261077 15242 raft_consensus.cc:1261] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Refusing update from remote peer 0c622bb34dcf4baaa83492150ca5044d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:18.261656 15592 consensus_queue.cc:1035] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:18.261945 15660 consensus_queue.cc:1035] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:18.313659 15078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:18.313978 15078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:18.317104 15078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: Sending DeleteTablet for 3 replicas of tablet 53da65e5ed224675821166d313490f6d
I20260928 15:44:18.317942 15218 tablet_service.cc:1514] Processing DeleteTablet for tablet 53da65e5ed224675821166d313490f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:18 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47264
I20260928 15:44:18.318344 15496 tablet_service.cc:1514] Processing DeleteTablet for tablet 53da65e5ed224675821166d313490f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:18 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47194
I20260928 15:44:18.318472 15666 tablet_replica.cc:330] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: stopping tablet replica
I20260928 15:44:18.318681 15667 tablet_replica.cc:330] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: stopping tablet replica
I20260928 15:44:18.318713 15666 raft_consensus.cc:2229] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:18.318892 15667 raft_consensus.cc:2229] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:18.318957 15666 raft_consensus.cc:2258] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:18.319116 15667 raft_consensus.cc:2258] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:18.319212 15078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 15:44:18.319334 15357 tablet_service.cc:1514] Processing DeleteTablet for tablet 53da65e5ed224675821166d313490f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:18 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:56012
I20260928 15:44:18.319737 15668 tablet_replica.cc:330] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: stopping tablet replica
I20260928 15:44:18.319963 15668 raft_consensus.cc:2229] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 1 LEADER]: Raft consensus shutting down.
W20260928 15:44:18.320063 15078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:18.320478 15668 raft_consensus.cc:2258] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:18.320922 15666 ts_tablet_manager.cc:1905] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:18.321753 15667 ts_tablet_manager.cc:1905] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:18.326274 15666 ts_tablet_manager.cc:1918] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:18.326444 15666 log.cc:1196] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/53da65e5ed224675821166d313490f6d
I20260928 15:44:18.327015 15666 ts_tablet_manager.cc:1939] T 53da65e5ed224675821166d313490f6d P 67e6e39328174accadf9d1990a77d42f: Deleting consensus metadata
I20260928 15:44:18.328267 15067 catalog_manager.cc:4841] TS 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285): tablet 53da65e5ed224675821166d313490f6d (table test-table [id=d48dc7dac7c749599c7b488773823cd7]) successfully deleted
I20260928 15:44:18.329121 15668 ts_tablet_manager.cc:1905] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:18.330773 15667 ts_tablet_manager.cc:1918] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:18.330948 15667 log.cc:1196] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/53da65e5ed224675821166d313490f6d
I20260928 15:44:18.331411 15667 ts_tablet_manager.cc:1939] T 53da65e5ed224675821166d313490f6d P ff611e9554fb41c381cea809b2e7fbe0: Deleting consensus metadata
I20260928 15:44:18.333141 15218 tablet_service.cc:1467] Processing CreateTablet for tablet b40d15a461bb4a279a6b2658a857c948 (DEFAULT_TABLE table=test-table [id=987372be6871418bbf60c1bed514cd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:18.333312 15067 catalog_manager.cc:4841] TS ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257): tablet 53da65e5ed224675821166d313490f6d (table test-table [id=d48dc7dac7c749599c7b488773823cd7]) successfully deleted
I20260928 15:44:18.333627 15218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40d15a461bb4a279a6b2658a857c948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:18.334012 15496 tablet_service.cc:1467] Processing CreateTablet for tablet b40d15a461bb4a279a6b2658a857c948 (DEFAULT_TABLE table=test-table [id=987372be6871418bbf60c1bed514cd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:18.334091 15357 tablet_service.cc:1467] Processing CreateTablet for tablet b40d15a461bb4a279a6b2658a857c948 (DEFAULT_TABLE table=test-table [id=987372be6871418bbf60c1bed514cd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:18.334437 15496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40d15a461bb4a279a6b2658a857c948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:18.334513 15357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40d15a461bb4a279a6b2658a857c948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:18.341310 15677 tablet_bootstrap.cc:492] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: Bootstrap starting.
I20260928 15:44:18.344470 15677 tablet_bootstrap.cc:654] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:18.344722 15679 tablet_bootstrap.cc:492] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: Bootstrap starting.
I20260928 15:44:18.347535 15679 tablet_bootstrap.cc:654] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:18.348979 15668 ts_tablet_manager.cc:1918] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:44:18.351774 15668 log.cc:1196] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/53da65e5ed224675821166d313490f6d
I20260928 15:44:18.352286 15678 tablet_bootstrap.cc:492] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: Bootstrap starting.
I20260928 15:44:18.354992 15678 tablet_bootstrap.cc:654] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:18.358757 15668 ts_tablet_manager.cc:1939] T 53da65e5ed224675821166d313490f6d P 0c622bb34dcf4baaa83492150ca5044d: Deleting consensus metadata
I20260928 15:44:18.360424 15065 catalog_manager.cc:4841] TS 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349): tablet 53da65e5ed224675821166d313490f6d (table test-table [id=d48dc7dac7c749599c7b488773823cd7]) successfully deleted
I20260928 15:44:18.361084 15678 tablet_bootstrap.cc:492] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: No bootstrap required, opened a new log
I20260928 15:44:18.361274 15678 ts_tablet_manager.cc:1397] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:44:18.361698 15677 tablet_bootstrap.cc:492] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: No bootstrap required, opened a new log
I20260928 15:44:18.361924 15677 ts_tablet_manager.cc:1397] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.015s
I20260928 15:44:18.362177 15678 raft_consensus.cc:348] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.362514 15678 raft_consensus.cc:374] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:18.362538 15679 tablet_bootstrap.cc:492] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: No bootstrap required, opened a new log
I20260928 15:44:18.362679 15678 raft_consensus.cc:729] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c622bb34dcf4baaa83492150ca5044d, State: Initialized, Role: FOLLOWER
I20260928 15:44:18.362701 15679 ts_tablet_manager.cc:1397] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.011s
I20260928 15:44:18.362787 15677 raft_consensus.cc:348] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.363070 15677 raft_consensus.cc:374] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:18.363201 15677 raft_consensus.cc:729] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e6e39328174accadf9d1990a77d42f, State: Initialized, Role: FOLLOWER
I20260928 15:44:18.363098 15678 consensus_queue.cc:260] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.363461 15679 raft_consensus.cc:348] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.363514 15677 consensus_queue.cc:260] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.363713 15679 raft_consensus.cc:374] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:18.363799 15679 raft_consensus.cc:729] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff611e9554fb41c381cea809b2e7fbe0, State: Initialized, Role: FOLLOWER
I20260928 15:44:18.364104 15679 consensus_queue.cc:260] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.364809 15677 ts_tablet_manager.cc:1428] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 15:44:18.367679 15679 ts_tablet_manager.cc:1428] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260928 15:44:18.368511 15678 ts_tablet_manager.cc:1428] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 15:44:18.539225 15592 raft_consensus.cc:482] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:18.539482 15592 raft_consensus.cc:504] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.540309 15592 leader_election.cc:288] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285), ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257)
I20260928 15:44:18.541098 15237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40d15a461bb4a279a6b2658a857c948" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f" is_pre_election: true
I20260928 15:44:18.541294 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40d15a461bb4a279a6b2658a857c948" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff611e9554fb41c381cea809b2e7fbe0" is_pre_election: true
I20260928 15:44:18.541589 15237 raft_consensus.cc:2454] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 0.
I20260928 15:44:18.541605 15516 raft_consensus.cc:2454] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 0.
I20260928 15:44:18.542179 15310 leader_election.cc:302] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c622bb34dcf4baaa83492150ca5044d, 67e6e39328174accadf9d1990a77d42f; no voters:
I20260928 15:44:18.542582 15592 raft_consensus.cc:2781] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:18.542726 15592 raft_consensus.cc:482] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:18.542901 15592 raft_consensus.cc:3037] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:18.547045 15592 raft_consensus.cc:504] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.548143 15592 leader_election.cc:288] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 election: Requested vote from peers 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285), ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257)
I20260928 15:44:18.549039 15516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40d15a461bb4a279a6b2658a857c948" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff611e9554fb41c381cea809b2e7fbe0"
I20260928 15:44:18.549003 15237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40d15a461bb4a279a6b2658a857c948" candidate_uuid: "0c622bb34dcf4baaa83492150ca5044d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e6e39328174accadf9d1990a77d42f"
I20260928 15:44:18.549350 15516 raft_consensus.cc:3037] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:18.549353 15237 raft_consensus.cc:3037] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:18.552243 15516 raft_consensus.cc:2454] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 1.
I20260928 15:44:18.552824 15310 leader_election.cc:302] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c622bb34dcf4baaa83492150ca5044d, ff611e9554fb41c381cea809b2e7fbe0; no voters:
I20260928 15:44:18.553175 15592 raft_consensus.cc:2781] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:18.553463 15592 raft_consensus.cc:686] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 1 LEADER]: Becoming Leader. State: Replica: 0c622bb34dcf4baaa83492150ca5044d, State: Running, Role: LEADER
I20260928 15:44:18.553671 15237 raft_consensus.cc:2454] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c622bb34dcf4baaa83492150ca5044d in term 1.
I20260928 15:44:18.553972 15592 consensus_queue.cc:237] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } }
I20260928 15:44:18.557761 15079 catalog_manager.cc:5495] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66). New cstate: current_term: 1 leader_uuid: "0c622bb34dcf4baaa83492150ca5044d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c622bb34dcf4baaa83492150ca5044d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 } health_report { overall_health: UNKNOWN } } }
I20260928 15:44:18.609308 15516 raft_consensus.cc:1261] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Refusing update from remote peer 0c622bb34dcf4baaa83492150ca5044d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:18.609519 15237 raft_consensus.cc:1261] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Refusing update from remote peer 0c622bb34dcf4baaa83492150ca5044d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:18.609973 15592 consensus_queue.cc:1035] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff611e9554fb41c381cea809b2e7fbe0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:18.610397 15647 consensus_queue.cc:1035] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e6e39328174accadf9d1990a77d42f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:44:18.637892 15645 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 15:44:20.229768 15079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:20.230034 15079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:20.234683 15079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5622c97ed6e4bfc9c977d3b00d23798: Sending DeleteTablet for 3 replicas of tablet b40d15a461bb4a279a6b2658a857c948
I20260928 15:44:20.235558 15218 tablet_service.cc:1514] Processing DeleteTablet for tablet b40d15a461bb4a279a6b2658a857c948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:20 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47264
I20260928 15:44:20.235625 15356 tablet_service.cc:1514] Processing DeleteTablet for tablet b40d15a461bb4a279a6b2658a857c948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:20 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:56012
I20260928 15:44:20.235723 15496 tablet_service.cc:1514] Processing DeleteTablet for tablet b40d15a461bb4a279a6b2658a857c948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:20 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47194
I20260928 15:44:20.236442 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15152
I20260928 15:44:20.248656 15720 tablet_replica.cc:330] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: stopping tablet replica
I20260928 15:44:20.248996 15720 raft_consensus.cc:2229] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:20.249377 15720 raft_consensus.cc:2258] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:20.250208 15722 tablet_replica.cc:330] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: stopping tablet replica
I20260928 15:44:20.250492 15722 raft_consensus.cc:2229] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:20.250830 15722 raft_consensus.cc:2258] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:20.251818 15721 tablet_replica.cc:330] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: stopping tablet replica
I20260928 15:44:20.252081 15721 raft_consensus.cc:2229] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:44:20.252323 15720 ts_tablet_manager.cc:1905] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:20.252513 15721 raft_consensus.cc:2258] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:20.253705 15721 ts_tablet_manager.cc:1905] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:20.260872 15722 ts_tablet_manager.cc:1905] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:20.261217 15720 ts_tablet_manager.cc:1918] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:44:20.261281 15721 ts_tablet_manager.cc:1918] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:44:20.261476 15720 log.cc:1196] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/b40d15a461bb4a279a6b2658a857c948
I20260928 15:44:20.261543 15721 log.cc:1196] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/b40d15a461bb4a279a6b2658a857c948
I20260928 15:44:20.262110 15720 ts_tablet_manager.cc:1939] T b40d15a461bb4a279a6b2658a857c948 P 67e6e39328174accadf9d1990a77d42f: Deleting consensus metadata
I20260928 15:44:20.263374 15721 ts_tablet_manager.cc:1939] T b40d15a461bb4a279a6b2658a857c948 P 0c622bb34dcf4baaa83492150ca5044d: Deleting consensus metadata
I20260928 15:44:20.263365 15067 catalog_manager.cc:4841] TS 67e6e39328174accadf9d1990a77d42f (127.13.5.65:34285): tablet b40d15a461bb4a279a6b2658a857c948 (table test-table [id=987372be6871418bbf60c1bed514cd6c]) successfully deleted
I20260928 15:44:20.265051 15065 catalog_manager.cc:4841] TS 0c622bb34dcf4baaa83492150ca5044d (127.13.5.66:43349): tablet b40d15a461bb4a279a6b2658a857c948 (table test-table [id=987372be6871418bbf60c1bed514cd6c]) successfully deleted
I20260928 15:44:20.274794 15722 ts_tablet_manager.cc:1918] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:44:20.275017 15722 log.cc:1196] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/b40d15a461bb4a279a6b2658a857c948
I20260928 15:44:20.275528 15722 ts_tablet_manager.cc:1939] T b40d15a461bb4a279a6b2658a857c948 P ff611e9554fb41c381cea809b2e7fbe0: Deleting consensus metadata
Sep 28 15:44:20 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:20.276849 15067 catalog_manager.cc:4841] TS ff611e9554fb41c381cea809b2e7fbe0 (127.13.5.67:33257): tablet b40d15a461bb4a279a6b2658a857c948 (table test-table [id=987372be6871418bbf60c1bed514cd6c]) successfully deleted
I20260928 15:44:20.285327 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:20.238384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.13.5.65:34285)
0928 15:44:20.260369 (+ 21985us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:20.260374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:20.260432 (+ 58us) server_negotiation.cc:407] Connection header received
0928 15:44:20.260496 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:20.260513 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:20.260599 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:20.260724 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:20.263056 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:20.264414 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:20.266310 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:20.266928 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:20.267075 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:20.278078 (+ 11003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:20.278097 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:20.278119 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:20.278161 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:20.282372 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:20.283246 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:20.283254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:20.283256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:20.283430 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:20.283770 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:20.283773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:20.283775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:20.284297 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:20.284641 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:20.284832 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 15:44:20.285009 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21800,"thread_start_us":21651,"threads_started":1}
W20260928 15:44:20.286201 15283 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41660
W20260928 15:44:20.286895 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15152: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:20.287160 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 15152
I20260928 15:44:20.307451 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15292
Sep 28 15:44:20 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:44:20.324590 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:20.308272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.13.5.66:43349)
0928 15:44:20.308785 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:20.308789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:20.308807 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:44:20.308877 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:20.308881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:20.308948 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:20.309041 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:20.310428 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:20.311301 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:20.312943 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:20.313364 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:20.313521 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:20.318007 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:20.318023 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:20.318040 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:20.318080 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:20.321559 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:20.322499 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:20.322507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:20.322509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:20.322653 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:20.323033 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:20.323038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:20.323040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:20.323482 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:20.323721 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:20.324063 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 15:44:20.324240 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":270,"threads_started":1}
W20260928 15:44:20.325433 15422 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54434
W20260928 15:44:20.326141 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15292: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:20.326303 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 15292
I20260928 15:44:20.358539 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15431
Sep 28 15:44:20 dist-test-slave-j968 krb5kdc[15036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610258, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:44:20.376258 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:20.359329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.13.5.67:33257)
0928 15:44:20.359832 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:20.359837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:20.359933 (+ 96us) server_negotiation.cc:407] Connection header received
0928 15:44:20.359993 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:20.359997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:20.360075 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:20.360173 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:20.361589 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:20.362296 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:20.363575 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:20.364068 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:20.364161 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:20.368816 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:20.368833 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:20.368851 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:20.368888 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:20.372314 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:20.373715 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:20.373723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:20.373725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:20.373891 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:20.374310 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:20.374314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:20.374316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:20.374729 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:20.374998 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:20.375299 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 15:44:20.375681 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
W20260928 15:44:20.377040 15561 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40986
W20260928 15:44:20.377677 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15431: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:20.377892 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 15431
I20260928 15:44:20.401361 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15049
I20260928 15:44:20.402711 15121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:20.528647 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 15049
2026-09-28T15:44:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8450 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:22 dist-test-slave-j968 krb5kdc[15730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:22 dist-test-slave-j968 krb5kdc[15730](info): set up 2 sockets
Sep 28 15:44:22 dist-test-slave-j968 krb5kdc[15730](info): commencing operation
krb5kdc: starting...
W20260928 15:44:23.151026 13333 mini_kdc.cc:121] Time spent starting KDC: real 2.577s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:23 dist-test-slave-j968 krb5kdc[15730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:23.352126 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:38871
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:38871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:23.682550 15743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:23.682860 15743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:23.682955 15743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:23.690327 15743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:23.690452 15743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:23.690498 15743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:23.690549 15743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:23.690596 15743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:23.701606 15743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:38871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:38871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:23.703171 15743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:23.704561 15743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:23.716908 15748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:23.716938 15749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:23.717002 15751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:23.717813 15743 server_base.cc:1034] running on GCE node
I20260928 15:44:23.718612 15743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:23.720182 15743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:23.721482 15743 hybrid_clock.cc:648] HybridClock initialized: now 1790610263721379 us; error 45 us; skew 500 ppm
I20260928 15:44:23.721848 15743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:23 dist-test-slave-j968 krb5kdc[15730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610263, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:23.725008 15743 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:23.726527 15743 webserver.cc:460] Webserver started at http://127.13.5.126:35277/ using document root <none> and password file <none>
I20260928 15:44:23.727219 15743 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:23.727357 15743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:23.727671 15743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:23.730693 15743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "cd88fde416df48dfa2f8320fa79f2dad"
format_stamp: "Formatted at 2026-09-28 15:44:23 on dist-test-slave-j968"
server_key: "40481472dc9ffb5f2670fb830697b2c4"
server_key_iv: "f883289b644eaa83d402bf9ed35c8e48"
server_key_version: "encryptionkey@0"
I20260928 15:44:23.731447 15743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "cd88fde416df48dfa2f8320fa79f2dad"
format_stamp: "Formatted at 2026-09-28 15:44:23 on dist-test-slave-j968"
server_key: "40481472dc9ffb5f2670fb830697b2c4"
server_key_iv: "f883289b644eaa83d402bf9ed35c8e48"
server_key_version: "encryptionkey@0"
I20260928 15:44:23.736780 15743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:44:23.740489 15758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:23.741999 15743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:23.742195 15743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "cd88fde416df48dfa2f8320fa79f2dad"
format_stamp: "Formatted at 2026-09-28 15:44:23 on dist-test-slave-j968"
server_key: "40481472dc9ffb5f2670fb830697b2c4"
server_key_iv: "f883289b644eaa83d402bf9ed35c8e48"
server_key_version: "encryptionkey@0"
I20260928 15:44:23.742400 15743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:23.790620 15743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:23.795477 15743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:23.795735 15743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:23.815089 15743 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:38871
I20260928 15:44:23.815111 15809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:38871 every 8 connection(s)
I20260928 15:44:23.816648 15743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:23.820712 15810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:23.821863 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 15743
I20260928 15:44:23.822110 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:23.822952 13333 external_mini_cluster.cc:1428] Setting key 6a623e58f6b5d1750c5ad1a92cbd98ee
I20260928 15:44:23.835846 15810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad: Bootstrap starting.
Sep 28 15:44:23 dist-test-slave-j968 krb5kdc[15730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:23.839712 15810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:23.840839 15810 log.cc:824] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:23.844508 15810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad: No bootstrap required, opened a new log
I20260928 15:44:23.848078 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:23.825207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.13.5.126:38871)
0928 15:44:23.826056 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:23.826072 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:23.826113 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:44:23.827767 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:23.827794 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:23.828292 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:23.828659 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:23.830549 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:23.832027 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:23.833848 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:23.834639 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:23.834796 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:23.839634 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:23.839643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:23.839658 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:23.839706 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:23.843400 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:23.844158 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:23.844166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:23.844174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:23.844389 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:23.844787 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:23.844795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:23.844801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:23.845567 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:23.845875 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:23.846305 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 15:44:23.846909 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":411,"threads_started":1}
I20260928 15:44:23.853731 15810 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd88fde416df48dfa2f8320fa79f2dad" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38871 } }
I20260928 15:44:23.854118 15810 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:23.854214 15810 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd88fde416df48dfa2f8320fa79f2dad, State: Initialized, Role: FOLLOWER
I20260928 15:44:23.854791 15810 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd88fde416df48dfa2f8320fa79f2dad" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38871 } }
I20260928 15:44:23.855013 15810 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:23.855165 15810 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:23.855347 15810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:23.857765 15810 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd88fde416df48dfa2f8320fa79f2dad" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38871 } }
I20260928 15:44:23.858455 15810 leader_election.cc:302] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd88fde416df48dfa2f8320fa79f2dad; no voters:
I20260928 15:44:23.859194 15810 leader_election.cc:288] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:23.859280 15815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:23.860062 15815 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [term 1 LEADER]: Becoming Leader. State: Replica: cd88fde416df48dfa2f8320fa79f2dad, State: Running, Role: LEADER
I20260928 15:44:23.860603 15815 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd88fde416df48dfa2f8320fa79f2dad" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38871 } }
I20260928 15:44:23.861366 15810 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:23.863466 15817 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd88fde416df48dfa2f8320fa79f2dad. Latest consensus state: current_term: 1 leader_uuid: "cd88fde416df48dfa2f8320fa79f2dad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd88fde416df48dfa2f8320fa79f2dad" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38871 } } }
I20260928 15:44:23.863952 15817 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:23.864729 15816 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd88fde416df48dfa2f8320fa79f2dad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd88fde416df48dfa2f8320fa79f2dad" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38871 } } }
I20260928 15:44:23.865031 15816 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:23.873585 15822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:23.878515 15822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:23.888001 15822 catalog_manager.cc:1347] Generated new cluster ID: 0e8294df56dd4f49a7fc4a0472e339b8
I20260928 15:44:23.888134 15822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:23.897002 15822 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:23.898846 15822 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:23.910334 15822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd88fde416df48dfa2f8320fa79f2dad: Generated new TSK 0
I20260928 15:44:23.911512 15822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:23.975996 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38871
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:24.435659 15836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:24.436046 15836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:24.436213 15836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:24.443576 15836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:24.443729 15836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:24.443988 15836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:24.454397 15836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:24.455770 15836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:24.457145 15836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:24.471002 15842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:24.471031 15841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:24.470966 15844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:24.471670 15836 server_base.cc:1034] running on GCE node
I20260928 15:44:24.472416 15836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:24.473438 15836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:24.474689 15836 hybrid_clock.cc:648] HybridClock initialized: now 1790610264474646 us; error 30 us; skew 500 ppm
I20260928 15:44:24.475032 15836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:24 dist-test-slave-j968 krb5kdc[15730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:24.478499 15836 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:24.480005 15836 webserver.cc:460] Webserver started at http://127.13.5.65:36761/ using document root <none> and password file <none>
I20260928 15:44:24.480715 15836 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:24.480857 15836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:24.481185 15836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:24.484293 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "3712259939644018a2b377bc19fd0a9e"
format_stamp: "Formatted at 2026-09-28 15:44:24 on dist-test-slave-j968"
server_key: "508a3666423619a2bcb6ecb24f4a817b"
server_key_iv: "3ff948f498e2f2cf6ba71d304a2c78d0"
server_key_version: "encryptionkey@0"
I20260928 15:44:24.485116 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "3712259939644018a2b377bc19fd0a9e"
format_stamp: "Formatted at 2026-09-28 15:44:24 on dist-test-slave-j968"
server_key: "508a3666423619a2bcb6ecb24f4a817b"
server_key_iv: "3ff948f498e2f2cf6ba71d304a2c78d0"
server_key_version: "encryptionkey@0"
I20260928 15:44:24.490499 15836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:44:24.494271 15851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:24.495754 15836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:44:24.495986 15836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "3712259939644018a2b377bc19fd0a9e"
format_stamp: "Formatted at 2026-09-28 15:44:24 on dist-test-slave-j968"
server_key: "508a3666423619a2bcb6ecb24f4a817b"
server_key_iv: "3ff948f498e2f2cf6ba71d304a2c78d0"
server_key_version: "encryptionkey@0"
I20260928 15:44:24.496199 15836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:24.509776 15836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:24.514379 15836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:24.514703 15836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:24.515689 15836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:24.517158 15836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:24.517261 15836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:24.517421 15836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:24.517493 15836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:24.550963 15836 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43757
I20260928 15:44:24.551054 15963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43757 every 8 connection(s)
I20260928 15:44:24.552554 15836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:24.562280 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 15836
I20260928 15:44:24.562580 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:24.563275 13333 external_mini_cluster.cc:1428] Setting key 7aa01c4c681c3388969cc6986560ab51
Sep 28 15:44:24 dist-test-slave-j968 krb5kdc[15730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:24.574829 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:24.555125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:34291 (local address 127.13.5.126:38871)
0928 15:44:24.555725 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:24.555729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:24.556499 (+ 770us) server_negotiation.cc:407] Connection header received
0928 15:44:24.558396 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:24.558402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:24.558511 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:24.558630 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:24.560248 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:24.561077 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:24.562800 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:24.563206 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:24.563345 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:24.567293 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:24.567303 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:24.567311 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:24.567359 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:24.571257 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:24.572251 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:24.572257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:24.572269 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:24.572400 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:24.572869 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:24.572875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:24.572880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:24.573186 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:24.573351 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:24.574161 (+ 810us) server_negotiation.cc:299] Negotiation successful
0928 15:44:24.574405 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":391,"threads_started":1}
I20260928 15:44:24.575953 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:24.555603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38871 (local address 127.13.5.65:34291)
0928 15:44:24.556341 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:24.556380 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:24.558112 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:24.558881 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:24.558889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:24.559414 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:24.560079 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:24.560099 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:24.561274 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:24.561278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:24.562649 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:24.562669 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:24.563452 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:24.563460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:24.563719 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:24.564428 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:24.564451 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:24.567069 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:24.571436 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:24.571444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:24.571459 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:24.572108 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:24.572523 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:24.572528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:24.572531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:24.572740 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:24.573324 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:24.573333 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:24.573772 (+ 439us) client_negotiation.cc:769] Sending connection context
0928 15:44:24.574053 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 15:44:24.574668 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":316,"threads_started":1}
I20260928 15:44:24.577827 15964 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38871
I20260928 15:44:24.578183 15964 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:24.579047 15964 heartbeater.cc:511] Master 127.13.5.126:38871 requested a full tablet report, sending...
I20260928 15:44:24.581693 15775 ts_manager.cc:194] Registered new tserver with Master: 3712259939644018a2b377bc19fd0a9e (127.13.5.65:43757)
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:44:24.584045 15775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:34291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:24.610767 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38871
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:24.944180 15970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:24.944505 15970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:24.944669 15970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:24.952209 15970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:24.952360 15970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:24.952529 15970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:24.962656 15970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:24.964075 15970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:24.965499 15970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:24.980724 15976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:24.980748 15975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:24.980942 15978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:24.981722 15970 server_base.cc:1034] running on GCE node
I20260928 15:44:24.982354 15970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:24.983294 15970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:24.984524 15970 hybrid_clock.cc:648] HybridClock initialized: now 1790610264984448 us; error 65 us; skew 500 ppm
I20260928 15:44:24.984869 15970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:24 dist-test-slave-j968 krb5kdc[15730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:24.988147 15970 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:24.989748 15970 webserver.cc:460] Webserver started at http://127.13.5.66:32845/ using document root <none> and password file <none>
I20260928 15:44:24.990489 15970 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:24.990636 15970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:24.990949 15970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:24.994143 15970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "4be39020f17a4da3af51b2db1ef23f5b"
format_stamp: "Formatted at 2026-09-28 15:44:24 on dist-test-slave-j968"
server_key: "2b10dd4d4c6cf0845f702b2aa3908a38"
server_key_iv: "2a8dec4047d01256101f57a723e9f670"
server_key_version: "encryptionkey@0"
I20260928 15:44:24.994982 15970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "4be39020f17a4da3af51b2db1ef23f5b"
format_stamp: "Formatted at 2026-09-28 15:44:24 on dist-test-slave-j968"
server_key: "2b10dd4d4c6cf0845f702b2aa3908a38"
server_key_iv: "2a8dec4047d01256101f57a723e9f670"
server_key_version: "encryptionkey@0"
I20260928 15:44:25.000628 15970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:44:25.004426 15985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:25.005890 15970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:44:25.006072 15970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "4be39020f17a4da3af51b2db1ef23f5b"
format_stamp: "Formatted at 2026-09-28 15:44:24 on dist-test-slave-j968"
server_key: "2b10dd4d4c6cf0845f702b2aa3908a38"
server_key_iv: "2a8dec4047d01256101f57a723e9f670"
server_key_version: "encryptionkey@0"
I20260928 15:44:25.006299 15970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:25.021323 15970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:25.025730 15970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:25.026068 15970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:25.027038 15970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:25.028455 15970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:25.028558 15970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:25.028714 15970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:25.028793 15970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:25.063206 15970 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:44457
I20260928 15:44:25.063251 16097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:44457 every 8 connection(s)
I20260928 15:44:25.064543 15970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:25.069813 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 15970
I20260928 15:44:25.070021 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:25.070568 13333 external_mini_cluster.cc:1428] Setting key 013af7676646daae755a010089baa012
Sep 28 15:44:25 dist-test-slave-j968 krb5kdc[15730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610264, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:25.088963 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:25.067278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51493 (local address 127.13.5.126:38871)
0928 15:44:25.067542 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:25.067548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:25.068766 (+ 1218us) server_negotiation.cc:407] Connection header received
0928 15:44:25.071296 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:25.071302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:25.071400 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:25.071515 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:25.073514 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:25.074273 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:25.076822 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:25.077247 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:25.077354 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:25.081557 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:25.081566 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:25.081575 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:25.081623 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:25.085387 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:25.086153 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:25.086161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:25.086168 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:25.086336 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:25.086871 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:25.086877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:25.086882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:25.087299 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:25.087489 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:25.088279 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 15:44:25.088553 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:44:25.089824 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:25.067785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38871 (local address 127.13.5.66:51493)
0928 15:44:25.068593 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:25.068641 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:25.070936 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:25.071812 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:25.071831 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:25.072503 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:25.073326 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:25.073353 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:25.074427 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:25.074436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:25.076671 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:25.076689 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:25.077479 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:25.077488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:25.077816 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:25.078685 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:25.078709 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:25.081275 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:25.085564 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:25.085571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:25.085584 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:25.086009 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:25.086491 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:25.086496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:25.086500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:25.086733 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:25.087453 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:25.087458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:25.087794 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 15:44:25.088132 (+ 338us) client_negotiation.cc:240] Negotiation successful
0928 15:44:25.088571 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":276,"threads_started":1}
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:44:25.091486 16098 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38871
I20260928 15:44:25.091849 16098 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:25.092694 16098 heartbeater.cc:511] Master 127.13.5.126:38871 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
I20260928 15:44:25.094393 15775 ts_manager.cc:194] Registered new tserver with Master: 4be39020f17a4da3af51b2db1ef23f5b (127.13.5.66:44457)
I20260928 15:44:25.095784 15775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:51493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:25.119376 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38871
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:25.445628 16103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:25.445993 16103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:25.446144 16103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:25.453573 16103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:25.453734 16103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:25.453960 16103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:25.464003 16103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:25.465391 16103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:25.466907 16103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:25.480911 16108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:25.480821 16111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:25.480947 16109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:25.481650 16103 server_base.cc:1034] running on GCE node
I20260928 15:44:25.482264 16103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:25.483228 16103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:25.484423 16103 hybrid_clock.cc:648] HybridClock initialized: now 1790610265484351 us; error 62 us; skew 500 ppm
I20260928 15:44:25.484830 16103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:25 dist-test-slave-j968 krb5kdc[15730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610265, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:25.492022 16103 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:25.493731 16103 webserver.cc:460] Webserver started at http://127.13.5.67:38459/ using document root <none> and password file <none>
I20260928 15:44:25.494477 16103 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:25.494626 16103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:25.494956 16103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:25.498013 16103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "37c38bf1ca9d4f9c9322c9ac666902a0"
format_stamp: "Formatted at 2026-09-28 15:44:25 on dist-test-slave-j968"
server_key: "2da03503a5d99e931170f6f69af9661c"
server_key_iv: "a262ce5b59d95176884fbf7beaa066b5"
server_key_version: "encryptionkey@0"
I20260928 15:44:25.498809 16103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "37c38bf1ca9d4f9c9322c9ac666902a0"
format_stamp: "Formatted at 2026-09-28 15:44:25 on dist-test-slave-j968"
server_key: "2da03503a5d99e931170f6f69af9661c"
server_key_iv: "a262ce5b59d95176884fbf7beaa066b5"
server_key_version: "encryptionkey@0"
I20260928 15:44:25.504307 16103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:44:25.507710 16118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:25.509188 16103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:25.509397 16103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "37c38bf1ca9d4f9c9322c9ac666902a0"
format_stamp: "Formatted at 2026-09-28 15:44:25 on dist-test-slave-j968"
server_key: "2da03503a5d99e931170f6f69af9661c"
server_key_iv: "a262ce5b59d95176884fbf7beaa066b5"
server_key_version: "encryptionkey@0"
I20260928 15:44:25.509639 16103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:25.529321 16103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:25.533385 16103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:25.533695 16103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:25.534652 16103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:25.536073 16103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:25.536177 16103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:25.536330 16103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:25.536407 16103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:25.572178 16103 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:45937
I20260928 15:44:25.572233 16230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:45937 every 8 connection(s)
I20260928 15:44:25.573575 16103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:25.578863 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 16103
I20260928 15:44:25.579097 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:25.579635 13333 external_mini_cluster.cc:1428] Setting key 078a1f298ff3b4b93b5adcdcb0d34c36
I20260928 15:44:25.587483 15964 heartbeater.cc:503] Master 127.13.5.126:38871 was elected leader, sending a full tablet report...
Sep 28 15:44:25 dist-test-slave-j968 krb5kdc[15730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610265, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:25.596819 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:25.576590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:34981 (local address 127.13.5.126:38871)
0928 15:44:25.576807 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:25.576812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:25.578007 (+ 1195us) server_negotiation.cc:407] Connection header received
0928 15:44:25.580703 (+ 2696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:25.580707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:25.580783 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:25.580865 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:25.583042 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:25.584104 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:25.585676 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:25.586161 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:25.586266 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:25.590403 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:25.590410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:25.590415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:25.590461 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:25.593221 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:25.594025 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:25.594030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:25.594034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:25.594214 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:25.594770 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:25.594776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:25.594779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:25.595206 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:25.595369 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:25.596315 (+ 946us) server_negotiation.cc:299] Negotiation successful
0928 15:44:25.596523 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:44:25.597797 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:25.577091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38871 (local address 127.13.5.67:34981)
0928 15:44:25.577840 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:25.577878 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:25.580195 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:25.581197 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:25.581210 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:25.581914 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:25.582776 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:25.582803 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:25.584262 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:25.584266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:25.585487 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:25.585537 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:25.586373 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:25.586378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:25.586623 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:25.587357 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:25.587380 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:25.590188 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:25.593360 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:25.593367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:25.593381 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:25.593886 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:25.594340 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:25.594347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:25.594352 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:25.594641 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:25.595370 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:25.595381 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:25.595916 (+ 535us) client_negotiation.cc:769] Sending connection context
0928 15:44:25.596223 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 15:44:25.596634 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":291,"threads_started":1}
I20260928 15:44:25.599522 16231 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38871
I20260928 15:44:25.599856 16231 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:25.600790 16231 heartbeater.cc:511] Master 127.13.5.126:38871 requested a full tablet report, sending...
I20260928 15:44:25.602524 15775 ts_manager.cc:194] Registered new tserver with Master: 37c38bf1ca9d4f9c9322c9ac666902a0 (127.13.5.67:45937)
I20260928 15:44:25.603399 15775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:34981
I20260928 15:44:25.608335 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:44:25.639084 15966 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:44:25.636528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.13.5.126:38871)
0928 15:44:25.636725 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:25.636730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:25.637233 (+ 503us) server_negotiation.cc:407] Connection header received
0928 15:44:25.637387 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:25.637391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:25.637458 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:25.637561 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:25.638886 (+ 1325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33458: BlockingRecv error: recv got EOF from 127.0.0.1:33458 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:44:25.642410 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15836
W20260928 15:44:25.645617 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15836: Not authorized: Client connection negotiation failed: client connection to 127.13.5.65:43757: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:44:25.645807 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 15836
I20260928 15:44:25.663002 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15970
W20260928 15:44:25.665762 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15970: Not authorized: Client connection negotiation failed: client connection to 127.13.5.66:44457: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:44:25.665982 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 15970
I20260928 15:44:25.682667 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16103
W20260928 15:44:25.685832 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16103: Not authorized: Client connection negotiation failed: client connection to 127.13.5.67:45937: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:44:25.685964 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 16103
I20260928 15:44:25.701839 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 15743
I20260928 15:44:25.703130 15805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:25.806810 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 15743
2026-09-28T15:44:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5275 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:25 dist-test-slave-j968 krb5kdc[16248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:25 dist-test-slave-j968 krb5kdc[16248](info): set up 2 sockets
Sep 28 15:44:25 dist-test-slave-j968 krb5kdc[16248](info): commencing operation
krb5kdc: starting...
W20260928 15:44:26.906988 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:27 dist-test-slave-j968 krb5kdc[16248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:27.104640 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:34529
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:34529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:27.430403 16261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:27.430768 16261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:27.430948 16261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:27.438108 16261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:27.438270 16261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:27.438360 16261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:27.438445 16261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:27.438544 16261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:27.448385 16261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:34529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:34529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:27.450029 16261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:27.451387 16261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:27.463344 16267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:27.463411 16269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:27.463510 16266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:27.464310 16261 server_base.cc:1034] running on GCE node
I20260928 15:44:27.465093 16261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:27.466490 16261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:27.467748 16261 hybrid_clock.cc:648] HybridClock initialized: now 1790610267467686 us; error 56 us; skew 500 ppm
I20260928 15:44:27.468158 16261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:27 dist-test-slave-j968 krb5kdc[16248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610267, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:27.471412 16261 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:27.473202 16261 webserver.cc:460] Webserver started at http://127.13.5.126:41583/ using document root <none> and password file <none>
I20260928 15:44:27.473951 16261 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:27.474116 16261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:27.474437 16261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:27.477553 16261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "8f4d9ca9f50c41768858604e80e974e7"
format_stamp: "Formatted at 2026-09-28 15:44:27 on dist-test-slave-j968"
server_key: "d39ce75fc73b465b2c9a32da17ed2704"
server_key_iv: "5f15bbb6c413e7bd91689c30e3344dfb"
server_key_version: "encryptionkey@0"
I20260928 15:44:27.478350 16261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "8f4d9ca9f50c41768858604e80e974e7"
format_stamp: "Formatted at 2026-09-28 15:44:27 on dist-test-slave-j968"
server_key: "d39ce75fc73b465b2c9a32da17ed2704"
server_key_iv: "5f15bbb6c413e7bd91689c30e3344dfb"
server_key_version: "encryptionkey@0"
I20260928 15:44:27.483652 16261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:44:27.487169 16276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:27.488586 16261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:27.488775 16261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "8f4d9ca9f50c41768858604e80e974e7"
format_stamp: "Formatted at 2026-09-28 15:44:27 on dist-test-slave-j968"
server_key: "d39ce75fc73b465b2c9a32da17ed2704"
server_key_iv: "5f15bbb6c413e7bd91689c30e3344dfb"
server_key_version: "encryptionkey@0"
I20260928 15:44:27.489012 16261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:27.506371 16261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:27.511319 16261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:27.511631 16261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:27.530058 16261 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:34529
I20260928 15:44:27.530092 16327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:34529 every 8 connection(s)
I20260928 15:44:27.531545 16261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:27.533787 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 16261
I20260928 15:44:27.534005 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:27.534825 13333 external_mini_cluster.cc:1428] Setting key f9b6cd75ed116c7106b018f03dc70d2e
I20260928 15:44:27.536514 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:44:27 dist-test-slave-j968 krb5kdc[16248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:27.550856 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7: Bootstrap starting.
I20260928 15:44:27.555299 16328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:27.556730 16328 log.cc:824] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:27.559598 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:27.536811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.13.5.126:34529)
0928 15:44:27.537616 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:27.537629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:27.537670 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:44:27.539271 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:27.539292 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:27.539687 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:27.540107 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:27.541964 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:27.543356 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:27.545166 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:27.545819 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:27.545951 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:27.550902 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:27.550914 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:27.550937 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:27.550987 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:27.554927 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:27.555703 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:27.555715 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:27.555729 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:27.555972 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:27.556346 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:27.556351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:27.556354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:27.557132 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:27.557498 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:27.557858 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 15:44:27.558327 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":384,"threads_started":1}
I20260928 15:44:27.560642 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7: No bootstrap required, opened a new log
I20260928 15:44:27.563609 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:44:27.571663 16328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d9ca9f50c41768858604e80e974e7" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34529 } }
I20260928 15:44:27.572147 16328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:27.572273 16328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4d9ca9f50c41768858604e80e974e7, State: Initialized, Role: FOLLOWER
I20260928 15:44:27.572891 16328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d9ca9f50c41768858604e80e974e7" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34529 } }
I20260928 15:44:27.573149 16328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:27.573285 16328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:27.573467 16328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:27.575670 16328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d9ca9f50c41768858604e80e974e7" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34529 } }
I20260928 15:44:27.576284 16328 leader_election.cc:302] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f4d9ca9f50c41768858604e80e974e7; no voters:
I20260928 15:44:27.576983 16328 leader_election.cc:288] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:27.577124 16334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:27.578305 16334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [term 1 LEADER]: Becoming Leader. State: Replica: 8f4d9ca9f50c41768858604e80e974e7, State: Running, Role: LEADER
I20260928 15:44:27.578931 16334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d9ca9f50c41768858604e80e974e7" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34529 } }
I20260928 15:44:27.579555 16328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:27.582216 16335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f4d9ca9f50c41768858604e80e974e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d9ca9f50c41768858604e80e974e7" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34529 } } }
I20260928 15:44:27.582671 16335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:27.584941 16336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f4d9ca9f50c41768858604e80e974e7. Latest consensus state: current_term: 1 leader_uuid: "8f4d9ca9f50c41768858604e80e974e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d9ca9f50c41768858604e80e974e7" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34529 } } }
I20260928 15:44:27.585378 16336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:27.586516 16339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:27.593938 16339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:27.603760 16339 catalog_manager.cc:1347] Generated new cluster ID: a0677eb7b3894d25addf300dcd0d424c
I20260928 15:44:27.603942 16339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:27.619000 16331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:27.609583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.13.5.126:34529)
0928 15:44:27.609766 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:27.609773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:27.610340 (+ 567us) server_negotiation.cc:407] Connection header received
0928 15:44:27.610530 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:27.610536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:27.610629 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:27.610738 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:27.612094 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:27.613224 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:27.615149 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:27.615801 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:27.615982 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:27.617255 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:27.617261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:27.617265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:27.618133 (+ 868us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:27.618182 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:27.618209 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:27.618336 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:27.618474 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 15:44:27.618571 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:44:27.619601 16339 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:27.621485 16339 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:27.654565 16339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f4d9ca9f50c41768858604e80e974e7: Generated new TSK 0
I20260928 15:44:27.655712 16339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:44:27.669416 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16261
I20260928 15:44:27.670614 16323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:27.765074 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 16261
2026-09-28T15:44:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1954 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:27 dist-test-slave-j968 krb5kdc[16362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:27 dist-test-slave-j968 krb5kdc[16362](info): set up 2 sockets
Sep 28 15:44:27 dist-test-slave-j968 krb5kdc[16362](info): commencing operation
krb5kdc: starting...
W20260928 15:44:28.858896 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:28 dist-test-slave-j968 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:29.060504 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:35535
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:29.379992 16375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:29.380288 16375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:29.380486 16375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:29.387665 16375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:29.387771 16375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:29.387866 16375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:29.387980 16375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:29.388055 16375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:29.398139 16375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:35535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:29.399544 16375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:29.400976 16375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:29.413031 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:29.413134 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:29.413406 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:29.413842 16375 server_base.cc:1034] running on GCE node
I20260928 15:44:29.414642 16375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:29.416137 16375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:29.417368 16375 hybrid_clock.cc:648] HybridClock initialized: now 1790610269417328 us; error 51 us; skew 500 ppm
I20260928 15:44:29.417713 16375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:29 dist-test-slave-j968 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610269, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:29.420830 16375 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:29.422355 16375 webserver.cc:460] Webserver started at http://127.13.5.126:35783/ using document root <none> and password file <none>
I20260928 15:44:29.423025 16375 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:29.423177 16375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:29.423480 16375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:29.426450 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "c96e19fc031a4d95913693d3d4544b0b"
format_stamp: "Formatted at 2026-09-28 15:44:29 on dist-test-slave-j968"
server_key: "799a9ed7334a06f2a7144cdc2f56ae6c"
server_key_iv: "e4aaf5c7528686e11f3131f402ae17ac"
server_key_version: "encryptionkey@0"
I20260928 15:44:29.427212 16375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "c96e19fc031a4d95913693d3d4544b0b"
format_stamp: "Formatted at 2026-09-28 15:44:29 on dist-test-slave-j968"
server_key: "799a9ed7334a06f2a7144cdc2f56ae6c"
server_key_iv: "e4aaf5c7528686e11f3131f402ae17ac"
server_key_version: "encryptionkey@0"
I20260928 15:44:29.432570 16375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 15:44:29.436005 16390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:29.437317 16375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:29.437502 16375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "c96e19fc031a4d95913693d3d4544b0b"
format_stamp: "Formatted at 2026-09-28 15:44:29 on dist-test-slave-j968"
server_key: "799a9ed7334a06f2a7144cdc2f56ae6c"
server_key_iv: "e4aaf5c7528686e11f3131f402ae17ac"
server_key_version: "encryptionkey@0"
I20260928 15:44:29.437708 16375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:29.479154 16375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:29.483776 16375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:29.484125 16375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:29.502835 16375 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:35535
I20260928 15:44:29.502853 16441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:35535 every 8 connection(s)
I20260928 15:44:29.504297 16375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:29.508194 16442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:29.509835 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 16375
I20260928 15:44:29.510030 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:29.510686 13333 external_mini_cluster.cc:1428] Setting key 53b0b4fd19602cd88d3e66f6057c8446
I20260928 15:44:29.519326 16442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b: Bootstrap starting.
I20260928 15:44:29.522465 16442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:29.523487 16442 log.cc:824] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:44:29 dist-test-slave-j968 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:29.526537 16442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b: No bootstrap required, opened a new log
I20260928 15:44:29.533051 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:29.512717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.13.5.126:35535)
0928 15:44:29.513514 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:29.513530 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:29.513580 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:44:29.515234 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:29.515256 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:29.515651 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:29.516063 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:29.517454 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:29.518746 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:29.520066 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:29.520754 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:29.520978 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:29.525374 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:29.525381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:29.525396 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:29.525434 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:29.528819 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:29.529536 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:29.529545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:29.529551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:29.529720 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:29.530128 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:29.530132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:29.530134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:29.530663 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:29.531011 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:29.531390 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 15:44:29.531797 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":341,"threads_started":1}
I20260928 15:44:29.535174 16442 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96e19fc031a4d95913693d3d4544b0b" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35535 } }
I20260928 15:44:29.535583 16442 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:29.535696 16442 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96e19fc031a4d95913693d3d4544b0b, State: Initialized, Role: FOLLOWER
I20260928 15:44:29.536394 16442 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96e19fc031a4d95913693d3d4544b0b" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35535 } }
I20260928 15:44:29.536654 16442 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:29.536798 16442 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:29.536972 16442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:29.539067 16442 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96e19fc031a4d95913693d3d4544b0b" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35535 } }
I20260928 15:44:29.539551 16442 leader_election.cc:302] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96e19fc031a4d95913693d3d4544b0b; no voters:
I20260928 15:44:29.540174 16442 leader_election.cc:288] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:29.540297 16447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:29.541760 16447 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [term 1 LEADER]: Becoming Leader. State: Replica: c96e19fc031a4d95913693d3d4544b0b, State: Running, Role: LEADER
I20260928 15:44:29.542385 16442 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:29.542387 16447 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96e19fc031a4d95913693d3d4544b0b" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35535 } }
I20260928 15:44:29.545686 16449 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96e19fc031a4d95913693d3d4544b0b. Latest consensus state: current_term: 1 leader_uuid: "c96e19fc031a4d95913693d3d4544b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96e19fc031a4d95913693d3d4544b0b" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35535 } } }
I20260928 15:44:29.546095 16449 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:29.545706 16448 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96e19fc031a4d95913693d3d4544b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96e19fc031a4d95913693d3d4544b0b" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35535 } } }
I20260928 15:44:29.546217 16448 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:29.547000 16456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:29.552515 16456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:29.561847 16456 catalog_manager.cc:1347] Generated new cluster ID: d78d6e64a21a4f018509723c9258b330
I20260928 15:44:29.561967 16456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:29.574296 16456 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:29.576237 16456 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:29.584551 16456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96e19fc031a4d95913693d3d4544b0b: Generated new TSK 0
I20260928 15:44:29.585661 16456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:29.644460 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:35535
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:29.977942 16468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:29.978289 16468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:29.978468 16468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:29.987912 16468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:29.988098 16468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:29.988253 16468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:29.998318 16468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:29.999639 16468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:30.001057 16468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:30.014926 16473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:30.014873 16474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:30.014873 16476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:30.015782 16468 server_base.cc:1034] running on GCE node
I20260928 15:44:30.016407 16468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:30.017277 16468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:30.018479 16468 hybrid_clock.cc:648] HybridClock initialized: now 1790610270018420 us; error 45 us; skew 500 ppm
I20260928 15:44:30.018805 16468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:30 dist-test-slave-j968 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610270, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:30.021935 16468 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:30.023380 16468 webserver.cc:460] Webserver started at http://127.13.5.65:34235/ using document root <none> and password file <none>
I20260928 15:44:30.024129 16468 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:30.024291 16468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:30.024549 16468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:30.027387 16468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "886f1bf189ff47ada3cb3849756f4114"
format_stamp: "Formatted at 2026-09-28 15:44:30 on dist-test-slave-j968"
server_key: "6ae64d41979626763786414da8b189ba"
server_key_iv: "0b29a03a95e18ec357fdabbe124e95eb"
server_key_version: "encryptionkey@0"
I20260928 15:44:30.028188 16468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "886f1bf189ff47ada3cb3849756f4114"
format_stamp: "Formatted at 2026-09-28 15:44:30 on dist-test-slave-j968"
server_key: "6ae64d41979626763786414da8b189ba"
server_key_iv: "0b29a03a95e18ec357fdabbe124e95eb"
server_key_version: "encryptionkey@0"
I20260928 15:44:30.033406 16468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:44:30.036934 16483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:30.038215 16468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:44:30.038383 16468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "886f1bf189ff47ada3cb3849756f4114"
format_stamp: "Formatted at 2026-09-28 15:44:30 on dist-test-slave-j968"
server_key: "6ae64d41979626763786414da8b189ba"
server_key_iv: "0b29a03a95e18ec357fdabbe124e95eb"
server_key_version: "encryptionkey@0"
I20260928 15:44:30.038554 16468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:30.052193 16468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:30.056721 16468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:30.057030 16468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:30.058053 16468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:30.059384 16468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:30.059489 16468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:30.059643 16468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:30.059719 16468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:30.093348 16468 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:40119
I20260928 15:44:30.093395 16595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:40119 every 8 connection(s)
I20260928 15:44:30.094656 16468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:30.095929 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 16468
I20260928 15:44:30.096138 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:30.097034 13333 external_mini_cluster.cc:1428] Setting key 40cc676bbdbc0c5c1dac6b67829ba390
Sep 28 15:44:30 dist-test-slave-j968 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610270, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:44:30.119143 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:30.097618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:54139 (local address 127.13.5.126:35535)
0928 15:44:30.098127 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:30.098135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:30.102194 (+ 4059us) server_negotiation.cc:407] Connection header received
0928 15:44:30.102268 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:30.102277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:30.102370 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:30.102486 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:30.104344 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:30.105359 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:30.107816 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:30.108282 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:30.108390 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:30.113076 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:30.113082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:30.113085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:30.113121 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:30.115760 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:30.116774 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:30.116779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:30.116781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:30.116899 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:30.117418 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:30.117421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:30.117423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:30.117744 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:30.117856 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:30.118628 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 15:44:30.118867 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":239,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
I20260928 15:44:30.120329 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:30.098242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:35535 (local address 127.13.5.65:54139)
0928 15:44:30.099128 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:30.099166 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:30.101821 (+ 2655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:30.102753 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:30.102762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:30.103279 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:30.104146 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:30.104173 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:30.105545 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:30.105550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:30.107635 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:30.107656 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:30.108541 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:30.108552 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:30.108902 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:30.109667 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:30.109704 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:30.112835 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:30.115950 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:30.115963 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:30.115986 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:30.116644 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:30.117021 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:30.117029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:30.117035 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:30.117305 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:30.117875 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:30.117886 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:30.118271 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 15:44:30.118507 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 15:44:30.118978 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":324,"threads_started":1}
I20260928 15:44:30.122216 16596 heartbeater.cc:347] Connected to a master server at 127.13.5.126:35535
I20260928 15:44:30.122581 16596 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:30.123399 16596 heartbeater.cc:511] Master 127.13.5.126:35535 requested a full tablet report, sending...
I20260928 15:44:30.125814 16407 ts_manager.cc:194] Registered new tserver with Master: 886f1bf189ff47ada3cb3849756f4114 (127.13.5.65:40119)
I20260928 15:44:30.127274 16407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:54139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:30.148618 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:35535
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:30.474300 16602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:30.474632 16602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:30.474782 16602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:30.482060 16602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:30.482164 16602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:30.482342 16602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:30.492632 16602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:30.493950 16602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:30.495291 16602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:30.509047 16607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:30.509186 16608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:30.509225 16610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:30.510272 16602 server_base.cc:1034] running on GCE node
I20260928 15:44:30.510968 16602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:30.512027 16602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:30.513248 16602 hybrid_clock.cc:648] HybridClock initialized: now 1790610270513181 us; error 63 us; skew 500 ppm
I20260928 15:44:30.513593 16602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:30 dist-test-slave-j968 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610270, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:30.523082 16602 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:30.524669 16602 webserver.cc:460] Webserver started at http://127.13.5.66:37733/ using document root <none> and password file <none>
I20260928 15:44:30.525324 16602 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:30.525425 16602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:30.525682 16602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:30.528678 16602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "ddc50f1138fd43d4a124d69673e1709c"
format_stamp: "Formatted at 2026-09-28 15:44:30 on dist-test-slave-j968"
server_key: "b32fe8cf244e7e38d98816ede9678287"
server_key_iv: "bb61fb5f05f3d72d9c271523c4f68e89"
server_key_version: "encryptionkey@0"
I20260928 15:44:30.529394 16602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "ddc50f1138fd43d4a124d69673e1709c"
format_stamp: "Formatted at 2026-09-28 15:44:30 on dist-test-slave-j968"
server_key: "b32fe8cf244e7e38d98816ede9678287"
server_key_iv: "bb61fb5f05f3d72d9c271523c4f68e89"
server_key_version: "encryptionkey@0"
I20260928 15:44:30.534672 16602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 15:44:30.538311 16617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:30.539636 16602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:44:30.539817 16602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "ddc50f1138fd43d4a124d69673e1709c"
format_stamp: "Formatted at 2026-09-28 15:44:30 on dist-test-slave-j968"
server_key: "b32fe8cf244e7e38d98816ede9678287"
server_key_iv: "bb61fb5f05f3d72d9c271523c4f68e89"
server_key_version: "encryptionkey@0"
I20260928 15:44:30.540086 16602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:30.554157 16602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:30.558646 16602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:30.558961 16602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:30.559967 16602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:30.561278 16602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:30.561367 16602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:30.561535 16602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:30.561597 16602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:30.596204 16602 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:45061
I20260928 15:44:30.596237 16729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:45061 every 8 connection(s)
I20260928 15:44:30.597575 16602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:30.608144 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 16602
I20260928 15:44:30.608377 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:30.608981 13333 external_mini_cluster.cc:1428] Setting key 9905c2e50e645412f3a23cc7c34da8ad
Sep 28 15:44:30 dist-test-slave-j968 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610270, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:30.619051 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:30.600371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:56237 (local address 127.13.5.126:35535)
0928 15:44:30.600601 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:30.600606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:30.601808 (+ 1202us) server_negotiation.cc:407] Connection header received
0928 15:44:30.603664 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:30.603670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:30.603771 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:30.603901 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:30.605452 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:30.606161 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:30.607737 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:30.608243 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:30.608386 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:30.612253 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:30.612262 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:30.612270 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:30.612316 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:30.615888 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:30.616617 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:30.616625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:30.616631 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:30.616796 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:30.617337 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:30.617340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:30.617342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:30.617625 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:30.617737 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:30.618643 (+ 906us) server_negotiation.cc:299] Negotiation successful
0928 15:44:30.618813 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 15:44:30.620675 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:30.600891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:35535 (local address 127.13.5.66:56237)
0928 15:44:30.601650 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:30.601692 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:30.603367 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:30.604147 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:30.604156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:30.604681 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:30.605278 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:30.605295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:30.606316 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:30.606327 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:30.607573 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:30.607607 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:30.608527 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:30.608537 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:30.608835 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:30.609598 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:30.609627 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:30.612022 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:30.616032 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:30.616039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:30.616052 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:30.616490 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:30.616908 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:30.616911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:30.616913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:30.617213 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:30.617754 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:30.617764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:30.618243 (+ 479us) client_negotiation.cc:769] Sending connection context
0928 15:44:30.618560 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 15:44:30.619168 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":281,"threads_started":1}
I20260928 15:44:30.622198 16730 heartbeater.cc:347] Connected to a master server at 127.13.5.126:35535
I20260928 15:44:30.622575 16730 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:30.623302 16730 heartbeater.cc:511] Master 127.13.5.126:35535 requested a full tablet report, sending...
I20260928 15:44:30.624909 16407 ts_manager.cc:194] Registered new tserver with Master: ddc50f1138fd43d4a124d69673e1709c (127.13.5.66:45061)
I20260928 15:44:30.625721 16407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:56237
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:30.656953 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:35535
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:30.987699 16735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:30.988023 16735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:30.988217 16735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:30.997121 16735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:30.997234 16735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:30.997349 16735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:31.008623 16735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:35535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:31.009938 16735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:31.011384 16735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:31.028168 16740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:31.028245 16743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:31.028194 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:31.029196 16735 server_base.cc:1034] running on GCE node
I20260928 15:44:31.029832 16735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:31.030855 16735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:31.032107 16735 hybrid_clock.cc:648] HybridClock initialized: now 1790610271032046 us; error 48 us; skew 500 ppm
I20260928 15:44:31.032485 16735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610271, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:31.035713 16735 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:31.037317 16735 webserver.cc:460] Webserver started at http://127.13.5.67:37941/ using document root <none> and password file <none>
I20260928 15:44:31.038161 16735 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:31.038309 16735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:31.038614 16735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:31.041764 16735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "8c8209f4cb674d3cafaef00ff46e9457"
format_stamp: "Formatted at 2026-09-28 15:44:31 on dist-test-slave-j968"
server_key: "9b429b4f2b73c4428c4e04d1b530fd10"
server_key_iv: "19c642dea6db4ecdef30004887968ccf"
server_key_version: "encryptionkey@0"
I20260928 15:44:31.042670 16735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "8c8209f4cb674d3cafaef00ff46e9457"
format_stamp: "Formatted at 2026-09-28 15:44:31 on dist-test-slave-j968"
server_key: "9b429b4f2b73c4428c4e04d1b530fd10"
server_key_iv: "19c642dea6db4ecdef30004887968ccf"
server_key_version: "encryptionkey@0"
I20260928 15:44:31.048466 16735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 15:44:31.051977 16750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:31.053390 16735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:31.053579 16735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "8c8209f4cb674d3cafaef00ff46e9457"
format_stamp: "Formatted at 2026-09-28 15:44:31 on dist-test-slave-j968"
server_key: "9b429b4f2b73c4428c4e04d1b530fd10"
server_key_iv: "19c642dea6db4ecdef30004887968ccf"
server_key_version: "encryptionkey@0"
I20260928 15:44:31.053812 16735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:31.075649 16735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:31.080570 16735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:31.080886 16735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:31.081905 16735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:31.083351 16735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:31.083457 16735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:31.083616 16735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:31.083689 16735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:31.119936 16735 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:37087
I20260928 15:44:31.119980 16862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:37087 every 8 connection(s)
I20260928 15:44:31.121381 16735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:31.126564 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 16735
I20260928 15:44:31.126760 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:31.127336 13333 external_mini_cluster.cc:1428] Setting key b168b1650159ee68a6642efb9f1ad73a
I20260928 15:44:31.131037 16596 heartbeater.cc:503] Master 127.13.5.126:35535 was elected leader, sending a full tablet report...
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610271, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:31.143537 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:31.124125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:57983 (local address 127.13.5.126:35535)
0928 15:44:31.124584 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:31.124589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:31.125725 (+ 1136us) server_negotiation.cc:407] Connection header received
0928 15:44:31.128313 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:31.128318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:31.128390 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:31.128542 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:31.130438 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.131443 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.133150 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.133572 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.133742 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:31.137207 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:31.137212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:31.137215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:31.137246 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:31.139947 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.140738 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.140742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.140745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.140876 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.141370 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.141375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.141380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.141754 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:31.141911 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:31.142908 (+ 997us) server_negotiation.cc:299] Negotiation successful
0928 15:44:31.143179 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
I20260928 15:44:31.144979 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:31.124747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:35535 (local address 127.13.5.67:57983)
0928 15:44:31.125557 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:31.125605 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:31.128000 (+ 2395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:31.128784 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:31.128799 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:31.129569 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:31.130242 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:31.130264 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.131591 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.131603 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:31.132990 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:31.133002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.133798 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.133806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:31.134055 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:31.134698 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:31.134722 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:31.137001 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:31.140096 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:31.140103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:31.140118 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:31.140630 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:31.141021 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:31.141026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:31.141029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:31.141229 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:31.141892 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:31.141919 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:31.142421 (+ 502us) client_negotiation.cc:769] Sending connection context
0928 15:44:31.142772 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 15:44:31.143517 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260928 15:44:31.146595 16863 heartbeater.cc:347] Connected to a master server at 127.13.5.126:35535
I20260928 15:44:31.146929 16863 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:31.147718 16863 heartbeater.cc:511] Master 127.13.5.126:35535 requested a full tablet report, sending...
I20260928 15:44:31.149240 16407 ts_manager.cc:194] Registered new tserver with Master: 8c8209f4cb674d3cafaef00ff46e9457 (127.13.5.67:37087)
I20260928 15:44:31.150077 16407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:57983
I20260928 15:44:31.156903 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610271, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610271, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:31.206063 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:31.191837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.13.5.126:35535)
0928 15:44:31.192066 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:31.192071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:31.192547 (+ 476us) server_negotiation.cc:407] Connection header received
0928 15:44:31.192677 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:31.192681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:31.192756 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:31.192876 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:31.194227 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.194968 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.196229 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.196628 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.196723 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:31.200555 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:31.200560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:31.200564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:31.200598 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:31.203090 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.203796 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.203801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.203803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.204051 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.204518 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.204523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.204525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.204895 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:31.205193 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:31.205502 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 15:44:31.205733 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 15:44:31.206614 16407 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56922
I20260928 15:44:31.208458 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16468
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610271, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:31.225899 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:31.209478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.13.5.65:40119)
0928 15:44:31.210208 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:31.210215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:31.210244 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:44:31.210300 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:31.210306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:31.210529 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:31.210699 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:31.212179 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.213201 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.214450 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.215047 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.215189 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:31.219593 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:31.219607 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:31.219624 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:31.219666 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:31.222879 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.223709 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.223717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.223719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.223848 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.224313 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.224317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.224319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.224707 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:31.224933 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:31.225260 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 15:44:31.225529 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":371,"threads_started":1}
W20260928 15:44:31.227368 16591 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49604
W20260928 15:44:31.228636 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16468: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:31.228840 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 16468
I20260928 15:44:31.245407 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16602
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610271, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:44:31.262262 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:31.246411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.13.5.66:45061)
0928 15:44:31.247068 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:31.247075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:31.247102 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:44:31.247164 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:31.247171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:31.247395 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:31.247584 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:31.248947 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.249963 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.251171 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.251741 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.251912 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:31.256019 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:31.256031 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:31.256056 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:31.256091 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:31.259177 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.259971 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.259983 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.259989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.260132 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.260534 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.260541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.260544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.260924 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:31.261262 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:31.261595 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 15:44:31.261891 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":364,"threads_started":1}
W20260928 15:44:31.263651 16725 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48572
W20260928 15:44:31.264925 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16602: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:31.265100 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 16602
I20260928 15:44:31.282425 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16735
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610271, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:44:31.300179 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:31.283480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.13.5.67:37087)
0928 15:44:31.284139 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:31.284147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:31.284177 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:44:31.284243 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:31.284249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:31.284471 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:31.284603 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:31.285895 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.286938 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.288600 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:31.289155 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:31.289253 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:31.293807 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:31.293818 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:31.293833 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:31.293866 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:31.297140 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.297902 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.297909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.297912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.298051 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:31.298493 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:31.298497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:31.298500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:31.298921 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:31.299175 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:31.299551 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 15:44:31.299774 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":323,"threads_started":1}
W20260928 15:44:31.301630 16858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51740
W20260928 15:44:31.302812 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16735: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:44:31.303005 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 16735
I20260928 15:44:31.319339 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16375
I20260928 15:44:31.320686 16437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:31.426770 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 16375
2026-09-28T15:44:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3665 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16881](info): set up 2 sockets
Sep 28 15:44:31 dist-test-slave-j968 krb5kdc[16881](info): commencing operation
krb5kdc: starting...
W20260928 15:44:32.535843 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:32 dist-test-slave-j968 krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:32.735553 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:40613
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:33.059384 16894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:33.059687 16894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:33.059850 16894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:33.067154 16894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:33.067256 16894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:33.067302 16894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:33.067411 16894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:33.067483 16894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:33.077466 16894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:40613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:33.078845 16894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:33.080272 16894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:33.092748 16902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:33.092694 16900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:33.092924 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:33.093506 16894 server_base.cc:1034] running on GCE node
I20260928 15:44:33.094316 16894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:33.095786 16894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:33.097050 16894 hybrid_clock.cc:648] HybridClock initialized: now 1790610273096981 us; error 49 us; skew 500 ppm
I20260928 15:44:33.097395 16894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:33 dist-test-slave-j968 krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610273, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:33.100473 16894 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:33.101902 16894 webserver.cc:460] Webserver started at http://127.13.5.126:34263/ using document root <none> and password file <none>
I20260928 15:44:33.102607 16894 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:33.102749 16894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:33.103075 16894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:33.106527 16894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "bee217694d6b41d38cc8a22a165f3955"
format_stamp: "Formatted at 2026-09-28 15:44:33 on dist-test-slave-j968"
server_key: "b3c41c309ebcc9d2ce7f83d9ddec2740"
server_key_iv: "63a4bfdcb8868cc7cdec1bc419f2e537"
server_key_version: "encryptionkey@0"
I20260928 15:44:33.107477 16894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "bee217694d6b41d38cc8a22a165f3955"
format_stamp: "Formatted at 2026-09-28 15:44:33 on dist-test-slave-j968"
server_key: "b3c41c309ebcc9d2ce7f83d9ddec2740"
server_key_iv: "63a4bfdcb8868cc7cdec1bc419f2e537"
server_key_version: "encryptionkey@0"
I20260928 15:44:33.113144 16894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:44:33.116755 16909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:33.118135 16894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:33.118336 16894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "bee217694d6b41d38cc8a22a165f3955"
format_stamp: "Formatted at 2026-09-28 15:44:33 on dist-test-slave-j968"
server_key: "b3c41c309ebcc9d2ce7f83d9ddec2740"
server_key_iv: "63a4bfdcb8868cc7cdec1bc419f2e537"
server_key_version: "encryptionkey@0"
I20260928 15:44:33.118549 16894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:33.145692 16894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:33.149953 16894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:33.150271 16894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:33.169514 16894 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:40613
I20260928 15:44:33.169553 16960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:40613 every 8 connection(s)
I20260928 15:44:33.171030 16894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:33.175102 16961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:33.175330 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 16894
I20260928 15:44:33.175565 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:33.176394 13333 external_mini_cluster.cc:1428] Setting key 99ee361ab496e3f8e455a9f3f7c60d6a
I20260928 15:44:33.189769 16961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955: Bootstrap starting.
Sep 28 15:44:33 dist-test-slave-j968 krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:33.193655 16961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:33.194944 16961 log.cc:824] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:33.198001 16961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955: No bootstrap required, opened a new log
I20260928 15:44:33.200977 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:33.178614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.13.5.126:40613)
0928 15:44:33.179473 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:33.179485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:33.179547 (+ 62us) server_negotiation.cc:407] Connection header received
0928 15:44:33.181669 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:33.181696 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:33.182168 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:33.182514 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:33.184400 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:33.185776 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:33.187611 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:33.188433 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:33.188589 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:33.193367 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:33.193376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:33.193392 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:33.193432 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:33.196738 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:33.197470 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:33.197477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:33.197483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:33.197643 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:33.198043 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:33.198048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:33.198051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:33.198649 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:33.198960 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:33.199396 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 15:44:33.199941 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":426,"threads_started":1}
I20260928 15:44:33.206032 16961 raft_consensus.cc:348] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee217694d6b41d38cc8a22a165f3955" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40613 } }
I20260928 15:44:33.206492 16961 raft_consensus.cc:374] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:33.206614 16961 raft_consensus.cc:729] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee217694d6b41d38cc8a22a165f3955, State: Initialized, Role: FOLLOWER
I20260928 15:44:33.207180 16961 consensus_queue.cc:260] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee217694d6b41d38cc8a22a165f3955" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40613 } }
I20260928 15:44:33.207459 16961 raft_consensus.cc:388] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:33.207619 16961 raft_consensus.cc:482] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:33.207798 16961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:33.210147 16961 raft_consensus.cc:504] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee217694d6b41d38cc8a22a165f3955" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40613 } }
I20260928 15:44:33.210668 16961 leader_election.cc:302] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bee217694d6b41d38cc8a22a165f3955; no voters:
I20260928 15:44:33.211326 16961 leader_election.cc:288] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:33.211499 16966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:33.212771 16966 raft_consensus.cc:686] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [term 1 LEADER]: Becoming Leader. State: Replica: bee217694d6b41d38cc8a22a165f3955, State: Running, Role: LEADER
I20260928 15:44:33.213450 16966 consensus_queue.cc:237] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee217694d6b41d38cc8a22a165f3955" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40613 } }
I20260928 15:44:33.213680 16961 sys_catalog.cc:564] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:33.216717 16968 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bee217694d6b41d38cc8a22a165f3955. Latest consensus state: current_term: 1 leader_uuid: "bee217694d6b41d38cc8a22a165f3955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee217694d6b41d38cc8a22a165f3955" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40613 } } }
I20260928 15:44:33.217207 16968 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:33.217936 16967 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bee217694d6b41d38cc8a22a165f3955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee217694d6b41d38cc8a22a165f3955" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40613 } } }
I20260928 15:44:33.218340 16967 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:33.219046 16972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:33.226068 16972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:33.235519 16972 catalog_manager.cc:1347] Generated new cluster ID: 8aee41bee3044cbaa8e0a273c16faf2e
I20260928 15:44:33.235639 16972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:33.247390 16972 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:33.248852 16972 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:33.262612 16972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bee217694d6b41d38cc8a22a165f3955: Generated new TSK 0
I20260928 15:44:33.263559 16972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:33.324481 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40613
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:33.676293 16987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:33.676573 16987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:33.676757 16987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:33.684711 16987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:33.684825 16987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:33.685035 16987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:33.695367 16987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:33.696714 16987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:33.698033 16987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:33.711086 16993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:33.711145 16995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:33.711052 16992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:33.712205 16987 server_base.cc:1034] running on GCE node
I20260928 15:44:33.712867 16987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:33.713908 16987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:33.715132 16987 hybrid_clock.cc:648] HybridClock initialized: now 1790610273715069 us; error 58 us; skew 500 ppm
I20260928 15:44:33.715458 16987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:33 dist-test-slave-j968 krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610273, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:33.718751 16987 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:33.720255 16987 webserver.cc:460] Webserver started at http://127.13.5.65:38929/ using document root <none> and password file <none>
I20260928 15:44:33.720996 16987 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:33.721134 16987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:33.721467 16987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:33.724690 16987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "6ab1e18cc1534c96a05fed97802295a4"
format_stamp: "Formatted at 2026-09-28 15:44:33 on dist-test-slave-j968"
server_key: "374110d49fce11882fb4290fb5ef940d"
server_key_iv: "6f18c5363fdbc21947ffc48674e40975"
server_key_version: "encryptionkey@0"
I20260928 15:44:33.725476 16987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "6ab1e18cc1534c96a05fed97802295a4"
format_stamp: "Formatted at 2026-09-28 15:44:33 on dist-test-slave-j968"
server_key: "374110d49fce11882fb4290fb5ef940d"
server_key_iv: "6f18c5363fdbc21947ffc48674e40975"
server_key_version: "encryptionkey@0"
I20260928 15:44:33.730726 16987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:44:33.734212 17002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:33.735622 16987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:44:33.735800 16987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "6ab1e18cc1534c96a05fed97802295a4"
format_stamp: "Formatted at 2026-09-28 15:44:33 on dist-test-slave-j968"
server_key: "374110d49fce11882fb4290fb5ef940d"
server_key_iv: "6f18c5363fdbc21947ffc48674e40975"
server_key_version: "encryptionkey@0"
I20260928 15:44:33.736086 16987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:33.754949 16987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:33.759446 16987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:33.759752 16987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:33.760761 16987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:33.762388 16987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:33.762465 16987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:33.762581 16987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:33.762643 16987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:33.795840 16987 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43381
I20260928 15:44:33.795933 17114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43381 every 8 connection(s)
I20260928 15:44:33.797263 16987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:33.807098 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 16987
I20260928 15:44:33.807345 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:33.808145 13333 external_mini_cluster.cc:1428] Setting key 1d6b3afeb5e43ba2059e03259fc5be27
Sep 28 15:44:33 dist-test-slave-j968 krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610273, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:33.818715 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:33.799916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:51755 (local address 127.13.5.126:40613)
0928 15:44:33.800380 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:33.800384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:33.801294 (+ 910us) server_negotiation.cc:407] Connection header received
0928 15:44:33.803207 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:33.803213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:33.803308 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:33.803429 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:33.805049 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:33.805790 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:33.807950 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:33.808442 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:33.808555 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:33.812615 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:33.812620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:33.812624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:33.812655 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:33.815390 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:33.816273 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:33.816277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:33.816280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:33.816439 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:33.816903 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:33.816912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:33.816914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:33.817189 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:33.817347 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:33.818085 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 15:44:33.818321 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":237,"threads_started":1}
I20260928 15:44:33.819729 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:33.800422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40613 (local address 127.13.5.65:51755)
0928 15:44:33.801127 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:33.801171 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:33.802917 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:33.803681 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:33.803689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:33.804276 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:33.804862 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:33.804879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:33.805941 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:33.805947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:33.807730 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:33.807747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:33.808691 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:33.808699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:33.808957 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:33.809711 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:33.809741 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:33.812368 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:33.815537 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:33.815547 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:33.815567 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:33.816101 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:33.816566 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:33.816573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:33.816577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:33.816778 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:33.817320 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:33.817328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:33.817704 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 15:44:33.817976 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 15:44:33.818565 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":262,"threads_started":1}
I20260928 15:44:33.821444 17115 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40613
I20260928 15:44:33.821815 17115 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:33.822659 17115 heartbeater.cc:511] Master 127.13.5.126:40613 requested a full tablet report, sending...
I20260928 15:44:33.824909 16926 ts_manager.cc:194] Registered new tserver with Master: 6ab1e18cc1534c96a05fed97802295a4 (127.13.5.65:43381)
I20260928 15:44:33.826438 16926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:51755
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:33.855958 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40613
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:34.184212 17121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:34.184474 17121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:34.184571 17121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:34.191697 17121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:34.191816 17121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:34.191958 17121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:34.201568 17121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:34.202852 17121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:34.204164 17121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:34.217624 17129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:34.217584 17126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:34.217635 17127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:34.218572 17121 server_base.cc:1034] running on GCE node
I20260928 15:44:34.219255 17121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:34.220337 17121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:34.221577 17121 hybrid_clock.cc:648] HybridClock initialized: now 1790610274221537 us; error 53 us; skew 500 ppm
I20260928 15:44:34.221913 17121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:34 dist-test-slave-j968 krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:34.225378 17121 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:34.226742 17121 webserver.cc:460] Webserver started at http://127.13.5.66:36505/ using document root <none> and password file <none>
I20260928 15:44:34.227403 17121 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:34.227504 17121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:34.227757 17121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:34.230993 17121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "9e0d99cd5ece40f5ac4cde51e7094a7c"
format_stamp: "Formatted at 2026-09-28 15:44:34 on dist-test-slave-j968"
server_key: "bce799caa28d32b3115b07c201160a76"
server_key_iv: "7fb6c3d20df64442653b3b9ed3b43330"
server_key_version: "encryptionkey@0"
I20260928 15:44:34.231725 17121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "9e0d99cd5ece40f5ac4cde51e7094a7c"
format_stamp: "Formatted at 2026-09-28 15:44:34 on dist-test-slave-j968"
server_key: "bce799caa28d32b3115b07c201160a76"
server_key_iv: "7fb6c3d20df64442653b3b9ed3b43330"
server_key_version: "encryptionkey@0"
I20260928 15:44:34.236788 17121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 15:44:34.240377 17136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:34.241868 17121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:44:34.242063 17121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "9e0d99cd5ece40f5ac4cde51e7094a7c"
format_stamp: "Formatted at 2026-09-28 15:44:34 on dist-test-slave-j968"
server_key: "bce799caa28d32b3115b07c201160a76"
server_key_iv: "7fb6c3d20df64442653b3b9ed3b43330"
server_key_version: "encryptionkey@0"
I20260928 15:44:34.242270 17121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:34.256929 17121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:34.261389 17121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:34.261695 17121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:34.262696 17121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:34.264289 17121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:34.264365 17121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:34.264454 17121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:34.264501 17121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:34.297534 17121 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:37043
I20260928 15:44:34.297582 17248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:37043 every 8 connection(s)
I20260928 15:44:34.298858 17121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:34.304908 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 17121
I20260928 15:44:34.305107 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:34.305665 13333 external_mini_cluster.cc:1428] Setting key 96cdb3e088a718993b712de82b3c205c
Sep 28 15:44:34 dist-test-slave-j968 krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:34.322618 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:34.301458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51359 (local address 127.13.5.126:40613)
0928 15:44:34.301649 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:34.301654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:34.302941 (+ 1287us) server_negotiation.cc:407] Connection header received
0928 15:44:34.305384 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:34.305389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:34.305471 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:34.305550 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:34.307550 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.308373 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.310947 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.311376 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.311515 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:34.316027 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:34.316033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:34.316037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:34.316069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:34.318903 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.319891 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.319898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.319902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.320089 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.320556 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.320586 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.320588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.320995 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:34.321177 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:34.322012 (+ 835us) server_negotiation.cc:299] Negotiation successful
0928 15:44:34.322249 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:44:34.323746 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:34.301878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40613 (local address 127.13.5.66:51359)
0928 15:44:34.302695 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:34.302786 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:34.305045 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:34.305866 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:34.305878 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:34.306536 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:34.307305 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:34.307329 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.308525 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.308530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:34.310781 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:34.310800 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.311602 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.311612 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:34.311988 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:34.312784 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:34.312818 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:34.315745 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:34.319198 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:34.319208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:34.319226 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:34.319738 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:34.320208 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:34.320213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:34.320217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:34.320429 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:34.321126 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:34.321134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:34.321604 (+ 470us) client_negotiation.cc:769] Sending connection context
0928 15:44:34.321928 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 15:44:34.322501 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":281,"threads_started":1}
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:44:34.325408 17249 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40613
I20260928 15:44:34.325716 17249 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:34.326509 17249 heartbeater.cc:511] Master 127.13.5.126:40613 requested a full tablet report, sending...
I20260928 15:44:34.328095 16926 ts_manager.cc:194] Registered new tserver with Master: 9e0d99cd5ece40f5ac4cde51e7094a7c (127.13.5.66:37043)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
I20260928 15:44:34.329281 16926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:51359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:34.353572 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40613
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:34.683756 17254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:34.684123 17254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:34.684221 17254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:34.691684 17254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:34.691788 17254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:34.691969 17254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:34.701892 17254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:34.703174 17254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:34.704833 17254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:34.719694 17259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:34.719736 17262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:34.719846 17260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:34.720871 17254 server_base.cc:1034] running on GCE node
I20260928 15:44:34.721678 17254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:34.722774 17254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:34.724043 17254 hybrid_clock.cc:648] HybridClock initialized: now 1790610274723987 us; error 49 us; skew 500 ppm
I20260928 15:44:34.724386 17254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:34 dist-test-slave-j968 krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:34.727579 17254 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:34.729220 17254 webserver.cc:460] Webserver started at http://127.13.5.67:36561/ using document root <none> and password file <none>
I20260928 15:44:34.729961 17254 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:34.730110 17254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:34.730435 17254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:34.733549 17254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "dd3f9de5382b406a8bee26321c67acd5"
format_stamp: "Formatted at 2026-09-28 15:44:34 on dist-test-slave-j968"
server_key: "fc35b07e346646f45f0c5ff2a3ced031"
server_key_iv: "1a542114eeda3f97cd08fb4a74eff24d"
server_key_version: "encryptionkey@0"
I20260928 15:44:34.734347 17254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "dd3f9de5382b406a8bee26321c67acd5"
format_stamp: "Formatted at 2026-09-28 15:44:34 on dist-test-slave-j968"
server_key: "fc35b07e346646f45f0c5ff2a3ced031"
server_key_iv: "1a542114eeda3f97cd08fb4a74eff24d"
server_key_version: "encryptionkey@0"
I20260928 15:44:34.739835 17254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:44:34.743785 17269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:34.745317 17254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:34.745569 17254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "dd3f9de5382b406a8bee26321c67acd5"
format_stamp: "Formatted at 2026-09-28 15:44:34 on dist-test-slave-j968"
server_key: "fc35b07e346646f45f0c5ff2a3ced031"
server_key_iv: "1a542114eeda3f97cd08fb4a74eff24d"
server_key_version: "encryptionkey@0"
I20260928 15:44:34.745767 17254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:34.763304 17254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:34.767803 17254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:34.768194 17254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:34.769136 17254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:34.770453 17254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:34.770526 17254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:34.770612 17254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:34.770659 17254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:34.804693 17254 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:45609
I20260928 15:44:34.804742 17381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:45609 every 8 connection(s)
I20260928 15:44:34.806016 17254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:34.813578 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 17254
I20260928 15:44:34.813783 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:34.814219 13333 external_mini_cluster.cc:1428] Setting key d61f9a541e4c6cde752675d889e4fa1b
Sep 28 15:44:34 dist-test-slave-j968 krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:34.827801 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:34.808707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:46669 (local address 127.13.5.126:40613)
0928 15:44:34.808925 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:34.808930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:34.810118 (+ 1188us) server_negotiation.cc:407] Connection header received
0928 15:44:34.811966 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:34.811972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:34.812071 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:34.812185 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:34.814135 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.815069 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.816693 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.817287 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.817421 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:34.821612 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:34.821617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:34.821620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:34.821653 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:34.824640 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.825450 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.825453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.825456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.825590 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.826042 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.826045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.826047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.826320 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:34.826432 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:34.827224 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 15:44:34.827454 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:44:34.829170 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:34.809210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40613 (local address 127.13.5.67:46669)
0928 15:44:34.809945 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:34.809984 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:34.811674 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:34.812436 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:34.812444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:34.813167 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:34.813957 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:34.813977 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.815223 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.815228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:34.816543 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:34.816555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.817530 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.817538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:34.817822 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:34.818733 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:34.818765 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:34.821380 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:34.824797 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:34.824804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:34.824821 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:34.825333 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:34.825699 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:34.825702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:34.825705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:34.825903 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:34.826461 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:34.826471 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:34.826851 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 15:44:34.827132 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 15:44:34.827709 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":287,"threads_started":1}
I20260928 15:44:34.829818 17115 heartbeater.cc:503] Master 127.13.5.126:40613 was elected leader, sending a full tablet report...
I20260928 15:44:34.830752 17382 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40613
I20260928 15:44:34.831070 17382 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:34.831833 17382 heartbeater.cc:511] Master 127.13.5.126:40613 requested a full tablet report, sending...
I20260928 15:44:34.833278 16926 ts_manager.cc:194] Registered new tserver with Master: dd3f9de5382b406a8bee26321c67acd5 (127.13.5.67:45609)
I20260928 15:44:34.834084 16926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:46669
I20260928 15:44:34.842373 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:44:34 dist-test-slave-j968 krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:44:34 dist-test-slave-j968 krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:34.891784 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:34.876108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.13.5.65:43381)
0928 15:44:34.877164 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:34.877171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:34.877201 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:44:34.877264 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:34.877271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:34.877503 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:34.877640 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:34.879027 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.880252 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.881483 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.882050 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.882157 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:34.886009 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:34.886022 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:34.886037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:34.886073 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:34.889028 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.889697 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.889705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.889707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.889855 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.890317 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.890321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.890323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.890692 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:34.890927 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:34.891202 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 15:44:34.891494 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":715,"threads_started":1}
Sep 28 15:44:34 dist-test-slave-j968 krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:34.910867 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:34.896357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.13.5.126:40613)
0928 15:44:34.896622 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:34.896628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:34.896805 (+ 177us) server_negotiation.cc:407] Connection header received
0928 15:44:34.896953 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:34.896956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:34.897022 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:34.897101 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:34.898382 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.899105 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.900541 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.901066 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.901162 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:34.905342 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:34.905347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:34.905350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:34.905386 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:34.908017 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.908685 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.908690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.908692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.908855 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.909285 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.909290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.909292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.909715 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:34.909921 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:34.910263 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 15:44:34.910536 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 15:44:34.911572 16926 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36322
I20260928 15:44:34.912786 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16987
I20260928 15:44:34.926913 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:34.913978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.13.5.65:43381)
0928 15:44:34.914248 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:34.914254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:34.914495 (+ 241us) server_negotiation.cc:407] Connection header received
0928 15:44:34.914559 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:34.914563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:34.914635 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:34.914843 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:34.916038 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.916739 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.917954 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:34.918402 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:34.918504 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:34.920842 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:34.920847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:34.920850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:34.920884 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:34.924181 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.924939 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.924944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.924947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.925081 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:34.925530 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:34.925534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:34.925537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:34.925815 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:34.926013 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:34.926287 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 15:44:34.926538 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 15:44:34.927536 17110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:35.012512 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 16987
I20260928 15:44:35.030906 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 17121
Sep 28 15:44:35 dist-test-slave-j968 krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:44:35.047310 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:35.031853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.13.5.66:37043)
0928 15:44:35.032415 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:35.032423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:35.032448 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:44:35.032511 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:35.032520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:35.032738 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:35.032911 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:35.034150 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:35.035141 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:35.036451 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:35.037015 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:35.037157 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:35.041413 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:35.041425 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:35.041441 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:35.041475 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:35.044440 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:35.045121 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:35.045129 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:35.045131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:35.045302 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:35.045744 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:35.045750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:35.045753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:35.046107 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:35.046428 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:35.046784 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 15:44:35.047013 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":215,"threads_started":1}
I20260928 15:44:35.048694 17244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:35.132653 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 17121
I20260928 15:44:35.150030 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 17254
Sep 28 15:44:35 dist-test-slave-j968 krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:44:35.167443 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:35.151030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.13.5.67:45609)
0928 15:44:35.151702 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:35.151709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:35.151739 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:44:35.151809 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:35.151815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:35.152114 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:35.152285 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:35.153602 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:35.154672 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:35.156054 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:35.156817 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:35.156956 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:35.161412 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:35.161425 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:35.161442 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:35.161476 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:35.164645 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:35.165372 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:35.165379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:35.165382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:35.165529 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:35.165904 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:35.165908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:35.165910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:35.166298 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:35.166577 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:35.166803 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 15:44:35.167012 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":322,"threads_started":1}
I20260928 15:44:35.168932 17377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:35.258713 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 17254
I20260928 15:44:35.275422 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 16894
I20260928 15:44:35.276909 16956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:35.382784 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 16894
2026-09-28T15:44:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3956 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:35 dist-test-slave-j968 krb5kdc[17401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:35 dist-test-slave-j968 krb5kdc[17401](info): set up 2 sockets
Sep 28 15:44:35 dist-test-slave-j968 krb5kdc[17401](info): commencing operation
krb5kdc: starting...
W20260928 15:44:36.491040 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:36 dist-test-slave-j968 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:36.692500 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:33527
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:33527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:37.030586 17414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:37.030946 17414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:37.031100 17414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:37.038640 17414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:37.038794 17414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:37.038905 17414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:37.039005 17414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:37.039091 17414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:37.049263 17414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:33527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:33527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:37.050689 17414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:37.052116 17414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:37.064150 17419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:37.064134 17420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:37.064759 17422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:37.065197 17414 server_base.cc:1034] running on GCE node
I20260928 15:44:37.065997 17414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:37.067481 17414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:37.068729 17414 hybrid_clock.cc:648] HybridClock initialized: now 1790610277068687 us; error 32 us; skew 500 ppm
I20260928 15:44:37.069111 17414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:37 dist-test-slave-j968 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:37.077562 17414 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:37.078933 17414 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:37.079074 17414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:37.079424 17414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:37.082579 17414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "d1d17de6f78b465a957eaef78a698f4d"
format_stamp: "Formatted at 2026-09-28 15:44:37 on dist-test-slave-j968"
server_key: "ec83fdab0b356bdd226e49701c567ec6"
server_key_iv: "e75f904800bad13af16b32cac140803c"
server_key_version: "encryptionkey@0"
I20260928 15:44:37.083388 17414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "d1d17de6f78b465a957eaef78a698f4d"
format_stamp: "Formatted at 2026-09-28 15:44:37 on dist-test-slave-j968"
server_key: "ec83fdab0b356bdd226e49701c567ec6"
server_key_iv: "e75f904800bad13af16b32cac140803c"
server_key_version: "encryptionkey@0"
I20260928 15:44:37.088630 17414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:44:37.092253 17428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:37.093827 17414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:37.094033 17414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "d1d17de6f78b465a957eaef78a698f4d"
format_stamp: "Formatted at 2026-09-28 15:44:37 on dist-test-slave-j968"
server_key: "ec83fdab0b356bdd226e49701c567ec6"
server_key_iv: "e75f904800bad13af16b32cac140803c"
server_key_version: "encryptionkey@0"
I20260928 15:44:37.094269 17414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:37.108088 17414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:37.112756 17414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:37.113059 17414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:37.132037 17414 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:33527
I20260928 15:44:37.132001 17479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:33527 every 8 connection(s)
I20260928 15:44:37.133054 17414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:37.136907 17480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:37.143356 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 17414
I20260928 15:44:37.143610 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:37.144517 13333 external_mini_cluster.cc:1428] Setting key c6a9d781211f41f70844635a367c54ec
I20260928 15:44:37.148972 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d: Bootstrap starting.
I20260928 15:44:37.152253 17480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:37.153347 17480 log.cc:824] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:37.156221 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d: No bootstrap required, opened a new log
Sep 28 15:44:37 dist-test-slave-j968 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:37.163775 17480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d17de6f78b465a957eaef78a698f4d" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33527 } }
I20260928 15:44:37.164335 17480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:37.164455 17480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d17de6f78b465a957eaef78a698f4d, State: Initialized, Role: FOLLOWER
I20260928 15:44:37.165112 17480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d17de6f78b465a957eaef78a698f4d" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33527 } }
I20260928 15:44:37.165441 17480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:37.165591 17480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:37.165783 17480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:37.168455 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:37.146898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.13.5.126:33527)
0928 15:44:37.147711 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:37.147724 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:37.147765 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:44:37.149684 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:37.149707 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:37.150111 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:37.150537 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:37.152229 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:37.153682 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:37.155000 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:37.155655 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:37.155866 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:37.159857 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:37.159934 (+ 77us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:37.159951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:37.160000 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:37.163247 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:37.164031 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:37.164039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:37.164046 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:37.164218 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:37.164849 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:37.164855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:37.164859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:37.165770 (+ 911us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:37.166147 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:37.166702 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 15:44:37.167214 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":309,"threads_started":1}
I20260928 15:44:37.168424 17480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d17de6f78b465a957eaef78a698f4d" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33527 } }
I20260928 15:44:37.168982 17480 leader_election.cc:302] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d17de6f78b465a957eaef78a698f4d; no voters:
I20260928 15:44:37.169685 17480 leader_election.cc:288] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:37.169878 17485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:37.171161 17485 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [term 1 LEADER]: Becoming Leader. State: Replica: d1d17de6f78b465a957eaef78a698f4d, State: Running, Role: LEADER
I20260928 15:44:37.172011 17480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:37.171906 17485 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d17de6f78b465a957eaef78a698f4d" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33527 } }
I20260928 15:44:37.174953 17487 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d17de6f78b465a957eaef78a698f4d. Latest consensus state: current_term: 1 leader_uuid: "d1d17de6f78b465a957eaef78a698f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d17de6f78b465a957eaef78a698f4d" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33527 } } }
I20260928 15:44:37.175343 17487 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:37.175609 17486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d17de6f78b465a957eaef78a698f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d17de6f78b465a957eaef78a698f4d" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33527 } } }
I20260928 15:44:37.176061 17486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:37.185153 17492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:37.191398 17492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:37.201149 17492 catalog_manager.cc:1347] Generated new cluster ID: 1dd6348b7def443fa11bddf198d4468e
I20260928 15:44:37.201300 17492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:37.218866 17492 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:37.220326 17492 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:37.228821 17492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d: Generated new TSK 0
I20260928 15:44:37.229764 17492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:37.294937 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33527
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:37.657050 17507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:37.657330 17507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:37.657559 17507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:37.664987 17507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:37.665133 17507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:37.665356 17507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:37.675666 17507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:37.677117 17507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:37.678495 17507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:37.691824 17513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:37.691896 17515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:37.691896 17512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:37.693049 17507 server_base.cc:1034] running on GCE node
I20260928 15:44:37.693696 17507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:37.694641 17507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:37.695945 17507 hybrid_clock.cc:648] HybridClock initialized: now 1790610277695830 us; error 44 us; skew 500 ppm
I20260928 15:44:37.696278 17507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:37 dist-test-slave-j968 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:37.699652 17507 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:37.700814 17507 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:37.700901 17507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:37.701256 17507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:37.704790 17507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "da4fbdf7252e436e9a319250d65b8502"
format_stamp: "Formatted at 2026-09-28 15:44:37 on dist-test-slave-j968"
server_key: "13f25b7837d4fa747b774912ac539425"
server_key_iv: "8957b53f2f7e9db4138ef87eed5510ee"
server_key_version: "encryptionkey@0"
I20260928 15:44:37.705539 17507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "da4fbdf7252e436e9a319250d65b8502"
format_stamp: "Formatted at 2026-09-28 15:44:37 on dist-test-slave-j968"
server_key: "13f25b7837d4fa747b774912ac539425"
server_key_iv: "8957b53f2f7e9db4138ef87eed5510ee"
server_key_version: "encryptionkey@0"
I20260928 15:44:37.710853 17507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:44:37.714509 17521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:37.716099 17507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:37.716259 17507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "da4fbdf7252e436e9a319250d65b8502"
format_stamp: "Formatted at 2026-09-28 15:44:37 on dist-test-slave-j968"
server_key: "13f25b7837d4fa747b774912ac539425"
server_key_iv: "8957b53f2f7e9db4138ef87eed5510ee"
server_key_version: "encryptionkey@0"
I20260928 15:44:37.716482 17507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:37.733829 17507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:37.738337 17507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:37.738626 17507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:37.739638 17507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:37.741086 17507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:37.741176 17507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:37.741322 17507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:37.741429 17507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:37.774721 17507 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:38613
I20260928 15:44:37.774803 17633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:38613 every 8 connection(s)
I20260928 15:44:37.775661 17507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:37.778422 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 17507
I20260928 15:44:37.778683 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:37.779361 13333 external_mini_cluster.cc:1428] Setting key 39d871521dfed05e515d63388679be0f
Sep 28 15:44:37 dist-test-slave-j968 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610277, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:37.801855 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:37.779971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:36035 (local address 127.13.5.126:33527)
0928 15:44:37.780513 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:37.780547 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:37.781105 (+ 558us) server_negotiation.cc:407] Connection header received
0928 15:44:37.784034 (+ 2929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:37.784038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:37.784113 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:37.784203 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:37.786541 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:37.787370 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:37.789582 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:37.790018 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:37.790128 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:37.794177 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:37.794186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:37.794194 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:37.794243 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:37.798027 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:37.798964 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:37.798973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:37.798979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:37.799158 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:37.799682 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:37.799690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:37.799696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:37.800162 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:37.800334 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:37.801133 (+ 799us) server_negotiation.cc:299] Negotiation successful
0928 15:44:37.801403 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":271,"threads_started":1}
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:44:37.802651 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:37.779332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33527 (local address 127.13.5.65:36035)
0928 15:44:37.780888 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:37.780966 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:37.783662 (+ 2696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:37.784500 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:37.784526 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:37.785384 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:37.786288 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:37.786316 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:37.787557 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:37.787565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:37.789381 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:37.789395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:37.790229 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:37.790233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:37.790431 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:37.791259 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:37.791282 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:37.793909 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:37.798242 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:37.798252 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:37.798267 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:37.798807 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:37.799321 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:37.799325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:37.799327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:37.799512 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:37.800300 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:37.800307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:37.800703 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 15:44:37.800950 (+ 247us) client_negotiation.cc:240] Negotiation successful
0928 15:44:37.801398 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1220,"thread_start_us":965,"threads_started":1}
I20260928 15:44:37.804342 17634 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33527
I20260928 15:44:37.804699 17634 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
I20260928 15:44:37.805673 17634 heartbeater.cc:511] Master 127.13.5.126:33527 requested a full tablet report, sending...
I20260928 15:44:37.808214 17445 ts_manager.cc:194] Registered new tserver with Master: da4fbdf7252e436e9a319250d65b8502 (127.13.5.65:38613)
I20260928 15:44:37.810410 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:36035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:37.832104 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33527
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:38.163251 17640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:38.163600 17640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:38.163769 17640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:38.171144 17640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:38.171295 17640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:38.171476 17640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:38.181594 17640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:38.182953 17640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:38.184291 17640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:38.198130 17645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:38.198136 17648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:38.198370 17646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:38.199072 17640 server_base.cc:1034] running on GCE node
I20260928 15:44:38.199703 17640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:38.200727 17640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:38.201925 17640 hybrid_clock.cc:648] HybridClock initialized: now 1790610278201873 us; error 49 us; skew 500 ppm
I20260928 15:44:38.202260 17640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:38 dist-test-slave-j968 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610278, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:38.205441 17640 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:38.206545 17640 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:38.206676 17640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:38.206984 17640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:38.210018 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c"
format_stamp: "Formatted at 2026-09-28 15:44:38 on dist-test-slave-j968"
server_key: "e55c88b4e8888a3d109e3514b4308bff"
server_key_iv: "406bdff88ee34830450dd96a5f055e1f"
server_key_version: "encryptionkey@0"
I20260928 15:44:38.210832 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c"
format_stamp: "Formatted at 2026-09-28 15:44:38 on dist-test-slave-j968"
server_key: "e55c88b4e8888a3d109e3514b4308bff"
server_key_iv: "406bdff88ee34830450dd96a5f055e1f"
server_key_version: "encryptionkey@0"
I20260928 15:44:38.216019 17640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 15:44:38.219353 17654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:38.220811 17640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:38.221001 17640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c"
format_stamp: "Formatted at 2026-09-28 15:44:38 on dist-test-slave-j968"
server_key: "e55c88b4e8888a3d109e3514b4308bff"
server_key_iv: "406bdff88ee34830450dd96a5f055e1f"
server_key_version: "encryptionkey@0"
I20260928 15:44:38.221236 17640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:38.246074 17640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:38.250653 17640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:38.250963 17640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:38.251984 17640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:38.253334 17640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:38.253460 17640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:38.253597 17640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:38.253685 17640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:38.287468 17640 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:39889
I20260928 15:44:38.287563 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:39889 every 8 connection(s)
I20260928 15:44:38.288739 17640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:38.292141 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 17640
I20260928 15:44:38.292382 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:38.292938 13333 external_mini_cluster.cc:1428] Setting key cf76a29ec2a2a0173ab41f3e9e1aa1d5
Sep 28 15:44:38 dist-test-slave-j968 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610278, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:38.311913 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.291316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:36613 (local address 127.13.5.126:33527)
0928 15:44:38.291607 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:38.291612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:38.292813 (+ 1201us) server_negotiation.cc:407] Connection header received
0928 15:44:38.294846 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:38.294854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:38.294942 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:38.295058 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:38.296670 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.297985 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.299725 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.300569 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.300730 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.305381 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:38.305387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:38.305390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:38.305655 (+ 265us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:38.308645 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:38.309467 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:38.309471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:38.309473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:38.309598 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:38.310059 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:38.310063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:38.310065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:38.310336 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:38.310514 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:38.311254 (+ 740us) server_negotiation.cc:299] Negotiation successful
0928 15:44:38.311506 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260928 15:44:38.313005 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.291704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33527 (local address 127.13.5.66:36613)
0928 15:44:38.292615 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:38.292664 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:38.294515 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:38.295263 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:38.295271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:38.295821 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:38.296465 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.296483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.298133 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.298154 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.299560 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.299572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.300792 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.300796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.300995 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.301919 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:38.301954 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:38.305099 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:38.308816 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:38.308826 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:38.308844 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:38.309311 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:38.309719 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:38.309724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:38.309728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:38.309920 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:38.310481 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:38.310489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:38.310868 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 15:44:38.311159 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 15:44:38.311748 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":441,"threads_started":1}
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:44:38.314682 17767 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33527
I20260928 15:44:38.314989 17767 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:38.315641 17767 heartbeater.cc:511] Master 127.13.5.126:33527 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
I20260928 15:44:38.317312 17445 ts_manager.cc:194] Registered new tserver with Master: ac0e0cf44e8e4fc196b9fea64e6eb95c (127.13.5.66:39889)
I20260928 15:44:38.318589 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:36613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:38.343509 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33527
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:38.674643 17772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:38.674958 17772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:38.675071 17772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:38.683205 17772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:38.683328 17772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:38.683466 17772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:38.694779 17772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:38.696234 17772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:38.697628 17772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:38.712752 17777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:38.712679 17780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:38.712695 17778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:38.713495 17772 server_base.cc:1034] running on GCE node
I20260928 15:44:38.714195 17772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:38.715270 17772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:38.716562 17772 hybrid_clock.cc:648] HybridClock initialized: now 1790610278716501 us; error 52 us; skew 500 ppm
I20260928 15:44:38.716917 17772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:38 dist-test-slave-j968 krb5kdc[17401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610278, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:38.720582 17772 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:38.722007 17772 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:38.722151 17772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:38.722514 17772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:38.726037 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "b5c421402e0d4952b650c8c1eccb2454"
format_stamp: "Formatted at 2026-09-28 15:44:38 on dist-test-slave-j968"
server_key: "f80e1f311ec3a73176e073ba31917376"
server_key_iv: "29482ccfc34e9ec63630306103d86729"
server_key_version: "encryptionkey@0"
I20260928 15:44:38.726970 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "b5c421402e0d4952b650c8c1eccb2454"
format_stamp: "Formatted at 2026-09-28 15:44:38 on dist-test-slave-j968"
server_key: "f80e1f311ec3a73176e073ba31917376"
server_key_iv: "29482ccfc34e9ec63630306103d86729"
server_key_version: "encryptionkey@0"
I20260928 15:44:38.732515 17772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:44:38.736302 17786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:38.737771 17772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:44:38.737980 17772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "b5c421402e0d4952b650c8c1eccb2454"
format_stamp: "Formatted at 2026-09-28 15:44:38 on dist-test-slave-j968"
server_key: "f80e1f311ec3a73176e073ba31917376"
server_key_iv: "29482ccfc34e9ec63630306103d86729"
server_key_version: "encryptionkey@0"
I20260928 15:44:38.738207 17772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:38.772396 17772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:38.777348 17772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:38.777699 17772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:38.778761 17772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:38.780220 17772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:38.780339 17772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:38.780467 17772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:38.780566 17772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:38.814442 17634 heartbeater.cc:503] Master 127.13.5.126:33527 was elected leader, sending a full tablet report...
I20260928 15:44:38.814560 17772 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:34665
I20260928 15:44:38.814579 17898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:34665 every 8 connection(s)
I20260928 15:44:38.815618 17772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:38.823714 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 17772
I20260928 15:44:38.823993 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:38.824496 13333 external_mini_cluster.cc:1428] Setting key d224351b34e98d1b5cca59901bbb595c
Sep 28 15:44:38 dist-test-slave-j968 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610278, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:38.836180 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.818142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:34141 (local address 127.13.5.126:33527)
0928 15:44:38.818577 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:38.818581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:38.819580 (+ 999us) server_negotiation.cc:407] Connection header received
0928 15:44:38.821580 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:38.821585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:38.821671 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:38.821787 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:38.823307 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.824185 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.825930 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.826411 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.826568 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.830647 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:38.830652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:38.830655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:38.830688 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:38.833189 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:38.833902 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:38.833914 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:38.833917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:38.834038 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:38.834441 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:38.834446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:38.834449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:38.834712 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:38.834863 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:38.835527 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 15:44:38.835752 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":239,"threads_started":1}
I20260928 15:44:38.837098 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.818676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33527 (local address 127.13.5.67:34141)
0928 15:44:38.819422 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:38.819470 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:38.821291 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:38.822029 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:38.822036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:38.822549 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:38.823123 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.823141 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.824339 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.824343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.825770 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.825782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.826657 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.826664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.826893 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.827703 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:38.827745 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:38.830341 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:38.833329 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:38.833337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:38.833365 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:38.833774 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:38.834147 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:38.834152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:38.834156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:38.834330 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:38.834834 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:38.834842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:38.835170 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 15:44:38.835441 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 15:44:38.835998 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":290,"threads_started":1}
I20260928 15:44:38.838531 17899 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33527
I20260928 15:44:38.838874 17899 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:38.839730 17899 heartbeater.cc:511] Master 127.13.5.126:33527 requested a full tablet report, sending...
I20260928 15:44:38.841185 17445 ts_manager.cc:194] Registered new tserver with Master: b5c421402e0d4952b650c8c1eccb2454 (127.13.5.67:34665)
I20260928 15:44:38.842021 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:34141
I20260928 15:44:38.852526 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:44:38.869177 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.857016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.13.5.126:33527)
0928 15:44:38.857250 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:38.857259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:38.857662 (+ 403us) server_negotiation.cc:407] Connection header received
0928 15:44:38.857799 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:38.857802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:38.857867 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:38.857948 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:38.859324 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.860184 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.861419 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.861814 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.861912 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.864246 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:38.864250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:38.864254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:38.864283 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:38.866702 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:38.867280 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:38.867284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:38.867286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:38.867395 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:38.867743 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:38.867746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:38.867748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:38.868141 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:38.868317 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:38.868629 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 15:44:38.868806 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:44:38.873914 17445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:38.881999 17445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:38.931377 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.895416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38613 (local address 127.0.0.1:44542)
0928 15:44:38.920520 (+ 25104us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:38.920581 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:38.920833 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:38.923920 (+ 3087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:38.923924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:38.923951 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:38.925042 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.925055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.926619 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.926628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.928842 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.928854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.930305 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.930311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.930505 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.930579 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 15:44:38.930709 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 15:44:38.930920 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24736,"mutex_wait_us":295,"thread_start_us":27805,"threads_started":1}
I20260928 15:44:38.932245 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.895710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.13.5.65:38613)
0928 15:44:38.922900 (+ 27190us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:38.922908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:38.922939 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:44:38.923024 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:38.923030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:38.923298 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:38.923613 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:38.925287 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.926494 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.929012 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.930103 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.930240 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.930485 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:38.931655 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0928 15:44:38.931892 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26991,"thread_start_us":26817,"threads_started":1}
I20260928 15:44:38.932734 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.896004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:34665 (local address 127.0.0.1:43992)
0928 15:44:38.922384 (+ 26380us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:38.922409 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:38.922705 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:38.923721 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:38.923732 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:38.923766 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:38.924764 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.924780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.927131 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.927138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.929712 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.929726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.931850 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.931855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.932062 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.932108 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 15:44:38.932190 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:44:38.932340 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26210,"mutex_wait_us":545,"thread_start_us":25055,"threads_started":1}
I20260928 15:44:38.932902 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.896138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:39889 (local address 127.0.0.1:38730)
0928 15:44:38.923064 (+ 26926us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:38.923077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:38.923207 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:38.924156 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:38.924160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:38.924186 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:38.925046 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.925055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.927019 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.927026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.929711 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:38.929726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.932152 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.932156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:38.932279 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.932325 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 15:44:38.932410 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:44:38.932545 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26790,"mutex_wait_us":124,"thread_start_us":25887,"threads_started":1}
I20260928 15:44:38.933462 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.896327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.13.5.66:39889)
0928 15:44:38.921144 (+ 24817us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:38.921153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:38.923190 (+ 2037us) server_negotiation.cc:407] Connection header received
0928 15:44:38.923446 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:38.923456 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:38.923706 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:38.923933 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:38.925245 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.926834 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.929889 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.931364 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.931798 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.932025 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:38.933124 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0928 15:44:38.933253 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24605,"thread_start_us":24369,"threads_started":1}
I20260928 15:44:38.934079 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:38.896669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.13.5.67:34665)
0928 15:44:38.919553 (+ 22884us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:38.919564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:38.922553 (+ 2989us) server_negotiation.cc:407] Connection header received
0928 15:44:38.922991 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:38.923000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:38.923299 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:38.923530 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:38.924981 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.926962 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.929890 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:38.931330 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:38.931464 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:38.931709 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:38.933080 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0928 15:44:38.933266 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22613,"thread_start_us":22912,"threads_started":1}
I20260928 15:44:38.936527 17569 tablet_service.cc:1467] Processing CreateTablet for tablet 730b40347b684d25948b92e67a55d379 (DEFAULT_TABLE table=test-table [id=dc774b21e4ae402f863924e830ba0f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:38.937269 17702 tablet_service.cc:1467] Processing CreateTablet for tablet 730b40347b684d25948b92e67a55d379 (DEFAULT_TABLE table=test-table [id=dc774b21e4ae402f863924e830ba0f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:38.937269 17834 tablet_service.cc:1467] Processing CreateTablet for tablet 730b40347b684d25948b92e67a55d379 (DEFAULT_TABLE table=test-table [id=dc774b21e4ae402f863924e830ba0f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:38.937863 17569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730b40347b684d25948b92e67a55d379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:38.938581 17834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730b40347b684d25948b92e67a55d379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:38.938578 17702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730b40347b684d25948b92e67a55d379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:38.949757 17918 tablet_bootstrap.cc:492] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Bootstrap starting.
I20260928 15:44:38.953121 17918 tablet_bootstrap.cc:654] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:38.953156 17920 tablet_bootstrap.cc:492] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Bootstrap starting.
I20260928 15:44:38.955070 17918 log.cc:824] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:38.955734 17919 tablet_bootstrap.cc:492] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Bootstrap starting.
I20260928 15:44:38.957640 17920 tablet_bootstrap.cc:654] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:38.958279 17918 tablet_bootstrap.cc:492] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: No bootstrap required, opened a new log
I20260928 15:44:38.958549 17918 ts_tablet_manager.cc:1397] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:44:38.958889 17920 log.cc:824] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:38.959466 17919 tablet_bootstrap.cc:654] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:38.960700 17919 log.cc:824] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:38.962060 17920 tablet_bootstrap.cc:492] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: No bootstrap required, opened a new log
I20260928 15:44:38.962338 17920 ts_tablet_manager.cc:1397] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:44:38.963820 17919 tablet_bootstrap.cc:492] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: No bootstrap required, opened a new log
I20260928 15:44:38.964097 17919 ts_tablet_manager.cc:1397] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260928 15:44:38.967222 17918 raft_consensus.cc:348] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:38.967613 17918 raft_consensus.cc:374] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:38.967729 17918 raft_consensus.cc:729] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4fbdf7252e436e9a319250d65b8502, State: Initialized, Role: FOLLOWER
I20260928 15:44:38.968466 17918 consensus_queue.cc:260] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:38.973327 17920 raft_consensus.cc:348] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:38.973886 17920 raft_consensus.cc:374] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:38.974045 17920 raft_consensus.cc:729] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0e0cf44e8e4fc196b9fea64e6eb95c, State: Initialized, Role: FOLLOWER
I20260928 15:44:38.974862 17920 consensus_queue.cc:260] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:38.974910 17919 raft_consensus.cc:348] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:38.975193 17918 ts_tablet_manager.cc:1428] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260928 15:44:38.975481 17919 raft_consensus.cc:374] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:38.975677 17919 raft_consensus.cc:729] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c421402e0d4952b650c8c1eccb2454, State: Initialized, Role: FOLLOWER
I20260928 15:44:38.976271 17767 heartbeater.cc:503] Master 127.13.5.126:33527 was elected leader, sending a full tablet report...
I20260928 15:44:38.976418 17919 consensus_queue.cc:260] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:38.976922 17920 ts_tablet_manager.cc:1428] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 15:44:38.978157 17899 heartbeater.cc:503] Master 127.13.5.126:33527 was elected leader, sending a full tablet report...
I20260928 15:44:38.978902 17919 ts_tablet_manager.cc:1428] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260928 15:44:39.028170 17635 tablet.cc:2369] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:39.040686 17768 tablet.cc:2369] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:39.067552 17900 tablet.cc:2369] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:44:39.188256 17926 raft_consensus.cc:482] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:39.188620 17926 raft_consensus.cc:504] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:39.190500 17926 leader_election.cc:288] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da4fbdf7252e436e9a319250d65b8502 (127.13.5.65:38613), ac0e0cf44e8e4fc196b9fea64e6eb95c (127.13.5.66:39889)
I20260928 15:44:39.196915 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:39.190991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:40527 (local address 127.13.5.65:38613)
0928 15:44:39.191190 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:39.191197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:39.191432 (+ 235us) server_negotiation.cc:407] Connection header received
0928 15:44:39.191531 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:39.191536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:39.191643 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:39.191761 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:39.192233 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.193309 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.195177 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.196011 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.196154 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:39.196258 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:39.196521 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 15:44:39.196649 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:44:39.196938 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:39.190783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38613 (local address 127.13.5.67:40527)
0928 15:44:39.191170 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:39.191191 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:39.191346 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:39.191814 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:39.191821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:39.191848 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:39.192051 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:39.192060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.193472 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.193477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:39.195014 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:39.195030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.196172 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.196178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:39.196355 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:39.196380 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 15:44:39.196503 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 15:44:39.196651 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":39}
I20260928 15:44:39.197245 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:39.191120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:59839 (local address 127.13.5.66:39889)
0928 15:44:39.191356 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:39.191365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:39.191747 (+ 382us) server_negotiation.cc:407] Connection header received
0928 15:44:39.191949 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:39.191955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:39.192026 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:39.192107 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:39.192633 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.193623 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.195435 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.196274 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.196431 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:39.196531 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:39.196821 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 15:44:39.196948 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:44:39.197687 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:39.190947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:39889 (local address 127.13.5.67:59839)
0928 15:44:39.191635 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:39.191648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:39.191782 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:39.192150 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:39.192157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:39.192183 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:39.192473 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:39.192481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.193782 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.193788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:39.195269 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:39.195282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.196509 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.196515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:39.196667 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:39.196690 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:44:39.196794 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:44:39.196925 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":44,"thread_start_us":382,"threads_started":1}
I20260928 15:44:39.198160 17589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730b40347b684d25948b92e67a55d379" candidate_uuid: "b5c421402e0d4952b650c8c1eccb2454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4fbdf7252e436e9a319250d65b8502" is_pre_election: true
I20260928 15:44:39.198779 17589 raft_consensus.cc:2454] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c421402e0d4952b650c8c1eccb2454 in term 0.
I20260928 15:44:39.198846 17722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730b40347b684d25948b92e67a55d379" candidate_uuid: "b5c421402e0d4952b650c8c1eccb2454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" is_pre_election: true
I20260928 15:44:39.199437 17722 raft_consensus.cc:2454] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5c421402e0d4952b650c8c1eccb2454 in term 0.
I20260928 15:44:39.199508 17789 leader_election.cc:302] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c421402e0d4952b650c8c1eccb2454, da4fbdf7252e436e9a319250d65b8502; no voters:
I20260928 15:44:39.199966 17926 raft_consensus.cc:2781] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:39.200183 17926 raft_consensus.cc:482] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:39.200290 17926 raft_consensus.cc:3037] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:39.202513 17926 raft_consensus.cc:504] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:39.203122 17926 leader_election.cc:288] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [CANDIDATE]: Term 1 election: Requested vote from peers da4fbdf7252e436e9a319250d65b8502 (127.13.5.65:38613), ac0e0cf44e8e4fc196b9fea64e6eb95c (127.13.5.66:39889)
I20260928 15:44:39.203684 17589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730b40347b684d25948b92e67a55d379" candidate_uuid: "b5c421402e0d4952b650c8c1eccb2454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da4fbdf7252e436e9a319250d65b8502"
I20260928 15:44:39.203691 17722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730b40347b684d25948b92e67a55d379" candidate_uuid: "b5c421402e0d4952b650c8c1eccb2454" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c"
I20260928 15:44:39.203922 17589 raft_consensus.cc:3037] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:39.203938 17722 raft_consensus.cc:3037] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:39.206226 17589 raft_consensus.cc:2454] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c421402e0d4952b650c8c1eccb2454 in term 1.
I20260928 15:44:39.206234 17722 raft_consensus.cc:2454] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5c421402e0d4952b650c8c1eccb2454 in term 1.
I20260928 15:44:39.206671 17789 leader_election.cc:302] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5c421402e0d4952b650c8c1eccb2454, da4fbdf7252e436e9a319250d65b8502; no voters:
I20260928 15:44:39.206950 17926 raft_consensus.cc:2781] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:39.207393 17926 raft_consensus.cc:686] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 1 LEADER]: Becoming Leader. State: Replica: b5c421402e0d4952b650c8c1eccb2454, State: Running, Role: LEADER
I20260928 15:44:39.207988 17926 consensus_queue.cc:237] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } }
I20260928 15:44:39.212672 17445 catalog_manager.cc:5495] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5c421402e0d4952b650c8c1eccb2454 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "b5c421402e0d4952b650c8c1eccb2454" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5c421402e0d4952b650c8c1eccb2454" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 } health_report { overall_health: UNKNOWN } } }
I20260928 15:44:39.230789 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:39.225599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.13.5.67:34665)
0928 15:44:39.225874 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:39.225880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:39.225924 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:44:39.226195 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:39.226199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:39.226278 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:39.226401 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:39.226822 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.227552 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.228885 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.229269 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.229363 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:39.229846 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:39.230333 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:39.230440 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:39.230524 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 15:44:39.230590 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:44:39.239015 17722 raft_consensus.cc:1261] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 1 FOLLOWER]: Refusing update from remote peer b5c421402e0d4952b650c8c1eccb2454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:39.239101 17589 raft_consensus.cc:1261] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 1 FOLLOWER]: Refusing update from remote peer b5c421402e0d4952b650c8c1eccb2454: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:39.240075 17928 consensus_queue.cc:1035] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac0e0cf44e8e4fc196b9fea64e6eb95c" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:39.240490 17926 consensus_queue.cc:1035] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da4fbdf7252e436e9a319250d65b8502" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:39.252388 17932 mvcc.cc:205] Tried to move back new op lower bound from 7334339703753330688 to 7334339703639879680. Current Snapshot: MvccSnapshot[applied={T|T < 7334339703753330688}]
I20260928 15:44:39.266180 17444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:39.266526 17444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:39.270629 17444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1d17de6f78b465a957eaef78a698f4d: Sending DeleteTablet for 3 replicas of tablet 730b40347b684d25948b92e67a55d379
I20260928 15:44:39.271919 17834 tablet_service.cc:1514] Processing DeleteTablet for tablet 730b40347b684d25948b92e67a55d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:39 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:43992
I20260928 15:44:39.271909 17569 tablet_service.cc:1514] Processing DeleteTablet for tablet 730b40347b684d25948b92e67a55d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:39 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:44542
I20260928 15:44:39.272105 17702 tablet_service.cc:1514] Processing DeleteTablet for tablet 730b40347b684d25948b92e67a55d379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:39 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:38730
I20260928 15:44:39.273970 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 17507
I20260928 15:44:39.286540 17941 tablet_replica.cc:330] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: stopping tablet replica
I20260928 15:44:39.287209 17943 tablet_replica.cc:330] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: stopping tablet replica
I20260928 15:44:39.290985 17943 raft_consensus.cc:2229] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:39.291678 17941 raft_consensus.cc:2229] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:39.292273 17941 raft_consensus.cc:2258] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:39.292778 17943 raft_consensus.cc:2258] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:39.293360 17942 tablet_replica.cc:330] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: stopping tablet replica
I20260928 15:44:39.293742 17942 raft_consensus.cc:2229] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:44:39.294348 17942 raft_consensus.cc:2258] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:39.297122 17941 ts_tablet_manager.cc:1905] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:39.297147 17942 ts_tablet_manager.cc:1905] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:39.300674 17943 ts_tablet_manager.cc:1905] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:44:39 dist-test-slave-j968 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:39.305029 17942 ts_tablet_manager.cc:1918] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:39.305284 17942 log.cc:1196] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/730b40347b684d25948b92e67a55d379
I20260928 15:44:39.306028 17942 ts_tablet_manager.cc:1939] T 730b40347b684d25948b92e67a55d379 P b5c421402e0d4952b650c8c1eccb2454: Deleting consensus metadata
I20260928 15:44:39.308040 17430 catalog_manager.cc:4841] TS b5c421402e0d4952b650c8c1eccb2454 (127.13.5.67:34665): tablet 730b40347b684d25948b92e67a55d379 (table test-table [id=dc774b21e4ae402f863924e830ba0f0b]) successfully deleted
I20260928 15:44:39.313686 17943 ts_tablet_manager.cc:1918] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:39.313884 17943 log.cc:1196] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/730b40347b684d25948b92e67a55d379
I20260928 15:44:39.314363 17943 ts_tablet_manager.cc:1939] T 730b40347b684d25948b92e67a55d379 P ac0e0cf44e8e4fc196b9fea64e6eb95c: Deleting consensus metadata
I20260928 15:44:39.316007 17429 catalog_manager.cc:4841] TS ac0e0cf44e8e4fc196b9fea64e6eb95c (127.13.5.66:39889): tablet 730b40347b684d25948b92e67a55d379 (table test-table [id=dc774b21e4ae402f863924e830ba0f0b]) successfully deleted
I20260928 15:44:39.316825 17941 ts_tablet_manager.cc:1918] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:39.317080 17941 log.cc:1196] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/730b40347b684d25948b92e67a55d379
I20260928 15:44:39.317742 17941 ts_tablet_manager.cc:1939] T 730b40347b684d25948b92e67a55d379 P da4fbdf7252e436e9a319250d65b8502: Deleting consensus metadata
I20260928 15:44:39.319748 17431 catalog_manager.cc:4841] TS da4fbdf7252e436e9a319250d65b8502 (127.13.5.65:38613): tablet 730b40347b684d25948b92e67a55d379 (table test-table [id=dc774b21e4ae402f863924e830ba0f0b]) successfully deleted
I20260928 15:44:39.323734 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:39.275944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.13.5.65:38613)
0928 15:44:39.276133 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:39.276141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:39.280881 (+ 4740us) server_negotiation.cc:407] Connection header received
0928 15:44:39.281995 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:39.282004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:39.282146 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:39.282240 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:39.287584 (+ 5344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.288766 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.297900 (+ 9134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.298674 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.298774 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:39.317085 (+ 18311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:39.317100 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:39.317115 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:39.317180 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:39.320634 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:39.321724 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:39.321732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:39.321734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:39.321875 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:39.322288 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:39.322292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:39.322293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:39.322675 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:39.322900 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:39.323254 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 15:44:39.323425 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:44:39.324466 17629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:39.417587 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 17507
I20260928 15:44:39.436429 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 17640
Sep 28 15:44:39 dist-test-slave-j968 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:44:39.454142 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:39.437296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.13.5.66:39889)
0928 15:44:39.437523 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:39.437528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:39.437547 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:44:39.437709 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:39.437713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:39.437778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:39.437871 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:39.439481 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.440280 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.442425 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.442881 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.443030 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:39.447685 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:39.447702 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:39.447719 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:39.447764 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:39.450900 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:39.451763 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:39.451771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:39.451773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:39.452052 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:39.452519 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:39.452523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:39.452525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:39.452939 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:39.453178 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:39.453608 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 15:44:39.453775 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:44:39.454881 17762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:39.546103 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 17640
I20260928 15:44:39.564044 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 17772
Sep 28 15:44:39 dist-test-slave-j968 krb5kdc[17401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:44:39.582880 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:39.564915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.13.5.67:34665)
0928 15:44:39.565103 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:39.565108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:39.565165 (+ 57us) server_negotiation.cc:407] Connection header received
0928 15:44:39.565351 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:39.565355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:39.565419 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:39.565502 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:39.567211 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.568062 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.569884 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:39.570333 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:39.570430 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:39.576247 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:39.576261 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:39.576279 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:39.576338 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:39.579618 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:39.580539 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:39.580547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:39.580549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:39.580702 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:39.581189 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:39.581201 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:39.581204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:39.581651 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:39.581929 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:39.582295 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 15:44:39.582477 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:44:39.583689 17894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:39.708040 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 17772
I20260928 15:44:39.729817 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 17414
I20260928 15:44:39.731055 17475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:39.841199 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 17414
2026-09-28T15:44:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4459 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T15:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:39Z Disabled control of system clock
I20260928 15:44:39.912395 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:40563
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:41137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:40563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:44:40.243424 17952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:40.243714 17952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:40.243944 17952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:40.251375 17952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:40.251485 17952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:40.251569 17952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:40.251683 17952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:40.251762 17952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:40.261088 17952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:41137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:40563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:40563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:40.262434 17952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:40.264282 17952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:40.276328 17957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:40.276337 17960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:40.276356 17958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:40.277472 17952 server_base.cc:1034] running on GCE node
I20260928 15:44:40.278303 17952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:40.279947 17952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:40.281172 17952 hybrid_clock.cc:648] HybridClock initialized: now 1790610280281113 us; error 50 us; skew 500 ppm
I20260928 15:44:40.281615 17952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:44:40.284078 17952 webserver.cc:460] Webserver started at http://127.13.5.126:36903/ using document root <none> and password file <none>
I20260928 15:44:40.284788 17952 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:40.284900 17952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:40.285243 17952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:40.288337 17952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "c73c6073d24d4c0dbc7fe36824b5caa2"
format_stamp: "Formatted at 2026-09-28 15:44:40 on dist-test-slave-j968"
server_key: "dba6d0720da20a53c8942972c48a05f5"
server_key_iv: "a3a6c8f02e3c8139956dd0e13e8873ba"
server_key_version: "encryptionkey@0"
I20260928 15:44:40.289168 17952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "c73c6073d24d4c0dbc7fe36824b5caa2"
format_stamp: "Formatted at 2026-09-28 15:44:40 on dist-test-slave-j968"
server_key: "dba6d0720da20a53c8942972c48a05f5"
server_key_iv: "a3a6c8f02e3c8139956dd0e13e8873ba"
server_key_version: "encryptionkey@0"
I20260928 15:44:40.294273 17952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:44:40.297634 17966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:40.298969 17952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 15:44:40.299145 17952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "c73c6073d24d4c0dbc7fe36824b5caa2"
format_stamp: "Formatted at 2026-09-28 15:44:40 on dist-test-slave-j968"
server_key: "dba6d0720da20a53c8942972c48a05f5"
server_key_iv: "a3a6c8f02e3c8139956dd0e13e8873ba"
server_key_version: "encryptionkey@0"
I20260928 15:44:40.299362 17952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:40.305147 17952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:40.305922 17952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:40.306218 17952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:40.324663 17952 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:40563
I20260928 15:44:40.324694 18017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:40563 every 8 connection(s)
I20260928 15:44:40.326170 17952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:40.330047 18018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:40.331394 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 17952
I20260928 15:44:40.331601 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:40.332494 13333 external_mini_cluster.cc:1428] Setting key f18cfa5827882079e2be0358eea02fdf
I20260928 15:44:40.340919 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:40.334584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.13.5.126:40563)
0928 15:44:40.335613 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:40.335625 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:40.335671 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:44:40.337244 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:40.337268 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:40.337697 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:40.338092 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:40.339207 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:40.339213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:40.339225 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:40.339237 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:40.339286 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:40.339300 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:40.339526 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:40.339671 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 15:44:40.339997 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":361,"threads_started":1}
I20260928 15:44:40.343609 18018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2: Bootstrap starting.
I20260928 15:44:40.346808 18018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:40.347712 18018 log.cc:824] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:40.350446 18018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2: No bootstrap required, opened a new log
I20260928 15:44:40.357885 18018 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c6073d24d4c0dbc7fe36824b5caa2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40563 } }
I20260928 15:44:40.358284 18018 raft_consensus.cc:374] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:40.358398 18018 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73c6073d24d4c0dbc7fe36824b5caa2, State: Initialized, Role: FOLLOWER
I20260928 15:44:40.358908 18018 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c6073d24d4c0dbc7fe36824b5caa2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40563 } }
I20260928 15:44:40.359102 18018 raft_consensus.cc:388] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:40.359197 18018 raft_consensus.cc:482] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:40.359315 18018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:40.361533 18018 raft_consensus.cc:504] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c6073d24d4c0dbc7fe36824b5caa2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40563 } }
I20260928 15:44:40.361980 18018 leader_election.cc:302] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c73c6073d24d4c0dbc7fe36824b5caa2; no voters:
I20260928 15:44:40.362562 18018 leader_election.cc:288] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:40.362649 18023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:40.363412 18023 raft_consensus.cc:686] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [term 1 LEADER]: Becoming Leader. State: Replica: c73c6073d24d4c0dbc7fe36824b5caa2, State: Running, Role: LEADER
I20260928 15:44:40.364125 18023 consensus_queue.cc:237] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c6073d24d4c0dbc7fe36824b5caa2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40563 } }
I20260928 15:44:40.364926 18018 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:40.367113 18025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c73c6073d24d4c0dbc7fe36824b5caa2. Latest consensus state: current_term: 1 leader_uuid: "c73c6073d24d4c0dbc7fe36824b5caa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c6073d24d4c0dbc7fe36824b5caa2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40563 } } }
I20260928 15:44:40.367638 18025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:40.367306 18024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c73c6073d24d4c0dbc7fe36824b5caa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c6073d24d4c0dbc7fe36824b5caa2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40563 } } }
I20260928 15:44:40.367851 18024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:40.369683 18032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:40.374598 18032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:40.382596 18032 catalog_manager.cc:1347] Generated new cluster ID: 46a96c03594b47b1b3e0aa59910532ce
I20260928 15:44:40.382711 18032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:40.393949 18032 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:40.394358 18032 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:40.403452 18032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2: Generated new TSK 0
I20260928 15:44:40.404474 18032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:44:40.412200 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40563
--builtin_ntp_servers=127.13.5.84:41137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:44:40.755716 18042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:40.756071 18042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:40.756166 18042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:40.763471 18042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:40.763657 18042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:40.763947 18042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:40.774019 18042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:41137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:40.775328 18042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:40.777246 18042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:40.791154 18047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:40.791178 18048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:40.791169 18050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:40.792140 18042 server_base.cc:1034] running on GCE node
I20260928 15:44:40.792778 18042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:40.793794 18042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:40.795042 18042 hybrid_clock.cc:648] HybridClock initialized: now 1790610280794988 us; error 42 us; skew 500 ppm
I20260928 15:44:40.795368 18042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:44:40.797667 18042 webserver.cc:460] Webserver started at http://127.13.5.65:33641/ using document root <none> and password file <none>
I20260928 15:44:40.798372 18042 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:40.798509 18042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:40.798842 18042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:40.801740 18042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "923052f6d29745448c5e8f697ebea001"
format_stamp: "Formatted at 2026-09-28 15:44:40 on dist-test-slave-j968"
server_key: "c92847ed6086312c3c728f3ca7150cb3"
server_key_iv: "847f77de0fe37024f866549254cb2bec"
server_key_version: "encryptionkey@0"
I20260928 15:44:40.802539 18042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "923052f6d29745448c5e8f697ebea001"
format_stamp: "Formatted at 2026-09-28 15:44:40 on dist-test-slave-j968"
server_key: "c92847ed6086312c3c728f3ca7150cb3"
server_key_iv: "847f77de0fe37024f866549254cb2bec"
server_key_version: "encryptionkey@0"
I20260928 15:44:40.807682 18042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:44:40.811344 18056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:40.812685 18042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:40.812868 18042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "923052f6d29745448c5e8f697ebea001"
format_stamp: "Formatted at 2026-09-28 15:44:40 on dist-test-slave-j968"
server_key: "c92847ed6086312c3c728f3ca7150cb3"
server_key_iv: "847f77de0fe37024f866549254cb2bec"
server_key_version: "encryptionkey@0"
I20260928 15:44:40.813102 18042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:40.818846 18042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:40.819662 18042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:40.819998 18042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:40.820920 18042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:40.822361 18042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:40.822489 18042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:40.822634 18042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:40.822727 18042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:40.857625 18042 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:39463
I20260928 15:44:40.857676 18168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:39463 every 8 connection(s)
I20260928 15:44:40.859025 18042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:40.864979 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 18042
I20260928 15:44:40.865180 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:40.865922 13333 external_mini_cluster.cc:1428] Setting key e3026dc74aac1b061658a5168d3f2699
I20260928 15:44:40.868626 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:40.862017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:51819 (local address 127.13.5.126:40563)
0928 15:44:40.862528 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:40.862532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:40.863835 (+ 1303us) server_negotiation.cc:407] Connection header received
0928 15:44:40.865828 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:40.865833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:40.865951 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:40.866092 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:40.867675 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:40.867680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:40.867685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:40.867704 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:40.867759 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:40.867771 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:40.867925 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:40.868182 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 15:44:40.868265 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260928 15:44:40.869133 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40563
--builtin_ntp_servers=127.13.5.84:41137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 15:44:40.870247 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:40.862668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40563 (local address 127.13.5.65:51819)
0928 15:44:40.863673 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:40.863718 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:40.865555 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:40.866507 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:40.866515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:40.866613 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:40.867325 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:40.867328 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:40.867372 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:40.867489 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:40.867896 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:40.867908 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:40.867917 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:44:40.868931 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0928 15:44:40.869151 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":311,"threads_started":1}
I20260928 15:44:40.873133 18169 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40563
I20260928 15:44:40.873452 18169 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:40.874048 18169 heartbeater.cc:511] Master 127.13.5.126:40563 requested a full tablet report, sending...
I20260928 15:44:40.876223 17983 ts_manager.cc:194] Registered new tserver with Master: 923052f6d29745448c5e8f697ebea001 (127.13.5.65:39463)
W20260928 15:44:41.208662 18173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:41.208968 18173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:41.209137 18173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:41.216495 18173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:41.216642 18173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:41.216823 18173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:41.226259 18173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:41137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:41.227658 18173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:41.229529 18173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:41.243242 18178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:41.243268 18179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:41.243203 18181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:41.244026 18173 server_base.cc:1034] running on GCE node
I20260928 15:44:41.244724 18173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:41.245709 18173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:41.246966 18173 hybrid_clock.cc:648] HybridClock initialized: now 1790610281246892 us; error 44 us; skew 500 ppm
I20260928 15:44:41.247303 18173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:44:41.249650 18173 webserver.cc:460] Webserver started at http://127.13.5.66:42125/ using document root <none> and password file <none>
I20260928 15:44:41.250367 18173 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:41.250496 18173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:41.250828 18173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:41.253728 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "40d83db656324e98a1e4407ab7149d4d"
format_stamp: "Formatted at 2026-09-28 15:44:41 on dist-test-slave-j968"
server_key: "0f91660634163de96e408c119729287f"
server_key_iv: "fc60549573d50deec22b751fbecfd2a2"
server_key_version: "encryptionkey@0"
I20260928 15:44:41.254608 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "40d83db656324e98a1e4407ab7149d4d"
format_stamp: "Formatted at 2026-09-28 15:44:41 on dist-test-slave-j968"
server_key: "0f91660634163de96e408c119729287f"
server_key_iv: "fc60549573d50deec22b751fbecfd2a2"
server_key_version: "encryptionkey@0"
I20260928 15:44:41.259747 18173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:44:41.263365 18187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:41.264933 18173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:44:41.265172 18173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "40d83db656324e98a1e4407ab7149d4d"
format_stamp: "Formatted at 2026-09-28 15:44:41 on dist-test-slave-j968"
server_key: "0f91660634163de96e408c119729287f"
server_key_iv: "fc60549573d50deec22b751fbecfd2a2"
server_key_version: "encryptionkey@0"
I20260928 15:44:41.265408 18173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:41.271595 18173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:41.272429 18173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:41.272696 18173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:41.273602 18173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:41.275074 18173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:41.275267 18173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:41.275557 18173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:41.275669 18173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:41.310273 18173 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:33917
I20260928 15:44:41.310285 18299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:33917 every 8 connection(s)
I20260928 15:44:41.311669 18173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:41.320159 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 18173
I20260928 15:44:41.320343 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:41.320396 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.314397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:46111 (local address 127.13.5.126:40563)
0928 15:44:41.314585 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.314589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.316166 (+ 1577us) server_negotiation.cc:407] Connection header received
0928 15:44:41.317994 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.318000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.318090 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.318219 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.319603 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.319608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.319613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.319636 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.319676 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.319688 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.319828 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.320061 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.320136 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:44:41.320953 13333 external_mini_cluster.cc:1428] Setting key 25bb4c2c1e3c17c3446aa63bbd030255
I20260928 15:44:41.321503 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.314937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40563 (local address 127.13.5.66:46111)
0928 15:44:41.316010 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.316055 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.317711 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.318478 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.318487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.318586 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.319254 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.319256 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.319296 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.319422 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.319785 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.319796 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.319805 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.320151 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.320443 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":297,"threads_started":1}
I20260928 15:44:41.322944 18300 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40563
I20260928 15:44:41.323256 18300 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:41.323947 18300 heartbeater.cc:511] Master 127.13.5.126:40563 requested a full tablet report, sending...
I20260928 15:44:41.324066 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40563
--builtin_ntp_servers=127.13.5.84:41137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 15:44:41.325670 17983 ts_manager.cc:194] Registered new tserver with Master: 40d83db656324e98a1e4407ab7149d4d (127.13.5.66:33917)
W20260928 15:44:41.659551 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:41.659832 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:41.659993 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:41.667394 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:41.667500 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:41.667610 18303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:41.677255 18303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:41137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:41.678586 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:41.680437 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:41.693771 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:41.693850 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:41.693922 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:41.694712 18303 server_base.cc:1034] running on GCE node
I20260928 15:44:41.695340 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:41.696336 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:41.697587 18303 hybrid_clock.cc:648] HybridClock initialized: now 1790610281697522 us; error 54 us; skew 500 ppm
I20260928 15:44:41.697868 18303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:44:41.700227 18303 webserver.cc:460] Webserver started at http://127.13.5.67:35539/ using document root <none> and password file <none>
I20260928 15:44:41.700897 18303 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:41.700984 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:41.701380 18303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:41.704428 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "983cba850c0949fa83530a1f567b8897"
format_stamp: "Formatted at 2026-09-28 15:44:41 on dist-test-slave-j968"
server_key: "1683f0bb8dd8912adba937e50074a2e5"
server_key_iv: "6d6436494f31d370236a7ec4de0c45ef"
server_key_version: "encryptionkey@0"
I20260928 15:44:41.705245 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "983cba850c0949fa83530a1f567b8897"
format_stamp: "Formatted at 2026-09-28 15:44:41 on dist-test-slave-j968"
server_key: "1683f0bb8dd8912adba937e50074a2e5"
server_key_iv: "6d6436494f31d370236a7ec4de0c45ef"
server_key_version: "encryptionkey@0"
I20260928 15:44:41.710618 18303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:44:41.714278 18317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:41.715755 18303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:44:41.715956 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "983cba850c0949fa83530a1f567b8897"
format_stamp: "Formatted at 2026-09-28 15:44:41 on dist-test-slave-j968"
server_key: "1683f0bb8dd8912adba937e50074a2e5"
server_key_iv: "6d6436494f31d370236a7ec4de0c45ef"
server_key_version: "encryptionkey@0"
I20260928 15:44:41.716193 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:41.722352 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:41.723170 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:41.723454 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:41.724478 18303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:41.725996 18303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:41.726116 18303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:41.726250 18303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:41.726341 18303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:41.760154 18303 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:39101
I20260928 15:44:41.760241 18429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:39101 every 8 connection(s)
I20260928 15:44:41.761583 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:41.766373 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 18303
I20260928 15:44:41.766563 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:41.767015 13333 external_mini_cluster.cc:1428] Setting key 3ca9da91a7f2bb00f1831dcf2a5e88cf
I20260928 15:44:41.770805 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.764419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:51731 (local address 127.13.5.126:40563)
0928 15:44:41.764638 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.764642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.766165 (+ 1523us) server_negotiation.cc:407] Connection header received
0928 15:44:41.768209 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.768215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.768286 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.768412 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.770005 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.770012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.770019 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.770039 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.770080 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.770104 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.770268 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.770451 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.770523 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:44:41.771855 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.764990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40563 (local address 127.13.5.67:51731)
0928 15:44:41.765974 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.766038 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.767939 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.768687 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.768695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.768793 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.769707 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.769709 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.769755 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.769849 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.770227 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.770238 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.770247 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.770539 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.770837 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":341,"threads_started":1}
I20260928 15:44:41.773208 18430 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40563
I20260928 15:44:41.773515 18430 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:41.774125 18430 heartbeater.cc:511] Master 127.13.5.126:40563 requested a full tablet report, sending...
I20260928 15:44:41.775528 17983 ts_manager.cc:194] Registered new tserver with Master: 983cba850c0949fa83530a1f567b8897 (127.13.5.67:39101)
I20260928 15:44:41.782856 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:44:41.789875 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.787307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.13.5.126:40563)
0928 15:44:41.787590 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.787599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.788080 (+ 481us) server_negotiation.cc:407] Connection header received
0928 15:44:41.788277 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.788281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.788339 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.788437 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.789291 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.789295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.789298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.789314 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.789345 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.789357 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.789455 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.789517 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.789563 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:44:41.793678 17983 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:41.800459 17983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:41.819092 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.813565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33917 (local address 127.0.0.1:50034)
0928 15:44:41.814137 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.814159 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.814259 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.815373 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.815381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.815475 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.816499 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.816505 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.816539 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.816633 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.818198 (+ 1565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.818207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.818216 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.818381 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.818458 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":238,"threads_started":1}
I20260928 15:44:41.819988 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.813904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.13.5.66:33917)
0928 15:44:41.814719 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.814728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.814761 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:44:41.814847 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.814858 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.815143 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.816101 (+ 958us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.817879 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.817891 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.817908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.817936 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.817998 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.818035 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.818891 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.818984 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.819126 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":671,"threads_started":1}
I20260928 15:44:41.820454 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.812594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.13.5.67:39101)
0928 15:44:41.813417 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.813429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.813481 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:44:41.813558 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.813566 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.813851 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.814013 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.817240 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.817254 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.817276 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.817301 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.817384 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.817418 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.817630 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.819707 (+ 2077us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.819914 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":482,"threads_started":1}
I20260928 15:44:41.818712 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.812382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39101 (local address 127.0.0.1:51262)
0928 15:44:41.813204 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.813246 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.813387 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.815330 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.815342 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.815476 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.816826 (+ 1350us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.816832 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.816856 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.816918 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.817712 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.817728 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.817741 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.817954 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.818068 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":320,"threads_started":1}
I20260928 15:44:41.823930 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.814310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.0.0.1:38028)
0928 15:44:41.819365 (+ 5055us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.819390 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.819505 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.821751 (+ 2246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.821759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.821817 (+ 58us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.822921 (+ 1104us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.822926 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.822947 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.822997 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.823463 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.823468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.823474 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.823590 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.823648 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4926,"thread_start_us":8161,"threads_started":1}
I20260928 15:44:41.824734 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.814403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.13.5.65:39463)
0928 15:44:41.821307 (+ 6904us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.821313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.821338 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:44:41.821410 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.821415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.821577 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.821691 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.823209 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.823218 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.823231 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.823260 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.823323 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.823341 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.823482 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.823604 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.823700 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6743,"thread_start_us":6614,"threads_started":1}
I20260928 15:44:41.826768 18235 tablet_service.cc:1467] Processing CreateTablet for tablet 30ce17558a0f42b2a497c89d236b477a (DEFAULT_TABLE table=test-table [id=ed176b7b8ae04da19ca5614404b69f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:41.826771 18365 tablet_service.cc:1467] Processing CreateTablet for tablet 30ce17558a0f42b2a497c89d236b477a (DEFAULT_TABLE table=test-table [id=ed176b7b8ae04da19ca5614404b69f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:41.828217 18365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ce17558a0f42b2a497c89d236b477a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:41.828260 18235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ce17558a0f42b2a497c89d236b477a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:41.828361 18104 tablet_service.cc:1467] Processing CreateTablet for tablet 30ce17558a0f42b2a497c89d236b477a (DEFAULT_TABLE table=test-table [id=ed176b7b8ae04da19ca5614404b69f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:41.829671 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ce17558a0f42b2a497c89d236b477a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:41.841792 18448 tablet_bootstrap.cc:492] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Bootstrap starting.
I20260928 15:44:41.844326 18450 tablet_bootstrap.cc:492] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001: Bootstrap starting.
I20260928 15:44:41.845743 18448 tablet_bootstrap.cc:654] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:41.847764 18448 log.cc:824] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:41.847980 18450 tablet_bootstrap.cc:654] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:41.848949 18449 tablet_bootstrap.cc:492] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: Bootstrap starting.
I20260928 15:44:41.849086 18450 log.cc:824] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:41.850965 18448 tablet_bootstrap.cc:492] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: No bootstrap required, opened a new log
I20260928 15:44:41.851248 18448 ts_tablet_manager.cc:1397] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 15:44:41.852032 18449 tablet_bootstrap.cc:654] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:41.852144 18450 tablet_bootstrap.cc:492] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001: No bootstrap required, opened a new log
I20260928 15:44:41.852392 18450 ts_tablet_manager.cc:1397] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260928 15:44:41.853032 18449 log.cc:824] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:41.856453 18449 tablet_bootstrap.cc:492] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: No bootstrap required, opened a new log
I20260928 15:44:41.856714 18449 ts_tablet_manager.cc:1397] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260928 15:44:41.863114 18448 raft_consensus.cc:348] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.863121 18450 raft_consensus.cc:348] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.863584 18450 raft_consensus.cc:374] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:41.863575 18448 raft_consensus.cc:374] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:41.863678 18450 raft_consensus.cc:729] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923052f6d29745448c5e8f697ebea001, State: Initialized, Role: FOLLOWER
I20260928 15:44:41.863672 18448 raft_consensus.cc:729] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983cba850c0949fa83530a1f567b8897, State: Initialized, Role: FOLLOWER
I20260928 15:44:41.864434 18448 consensus_queue.cc:260] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.864434 18450 consensus_queue.cc:260] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.864993 18449 raft_consensus.cc:348] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.865419 18449 raft_consensus.cc:374] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:41.865510 18449 raft_consensus.cc:729] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d83db656324e98a1e4407ab7149d4d, State: Initialized, Role: FOLLOWER
I20260928 15:44:41.866079 18449 consensus_queue.cc:260] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.872567 18169 heartbeater.cc:503] Master 127.13.5.126:40563 was elected leader, sending a full tablet report...
I20260928 15:44:41.873566 18450 ts_tablet_manager.cc:1428] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001: Time spent starting tablet: real 0.021s user 0.014s sys 0.005s
I20260928 15:44:41.874003 18430 heartbeater.cc:503] Master 127.13.5.126:40563 was elected leader, sending a full tablet report...
I20260928 15:44:41.876032 18300 heartbeater.cc:503] Master 127.13.5.126:40563 was elected leader, sending a full tablet report...
I20260928 15:44:41.877774 18449 ts_tablet_manager.cc:1428] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260928 15:44:41.880532 18448 ts_tablet_manager.cc:1428] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Time spent starting tablet: real 0.029s user 0.017s sys 0.005s
I20260928 15:44:41.964051 18455 raft_consensus.cc:482] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:41.964440 18455 raft_consensus.cc:504] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.966126 18456 raft_consensus.cc:482] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:41.966439 18456 raft_consensus.cc:504] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.966621 18455 leader_election.cc:288] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983cba850c0949fa83530a1f567b8897 (127.13.5.67:39101), 40d83db656324e98a1e4407ab7149d4d (127.13.5.66:33917)
I20260928 15:44:41.970347 18456 leader_election.cc:288] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983cba850c0949fa83530a1f567b8897 (127.13.5.67:39101), 923052f6d29745448c5e8f697ebea001 (127.13.5.65:39463)
I20260928 15:44:41.971167 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.967052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:60015 (local address 127.13.5.67:39101)
0928 15:44:41.967268 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.967275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.967893 (+ 618us) server_negotiation.cc:407] Connection header received
0928 15:44:41.968181 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.968186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.968258 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.968365 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.969837 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.969846 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.969853 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.969879 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.969931 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.969956 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.970076 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.970812 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.970876 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:44:41.971285 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.968242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:35205 (local address 127.13.5.66:33917)
0928 15:44:41.968453 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.968459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.968682 (+ 223us) server_negotiation.cc:407] Connection header received
0928 15:44:41.968974 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.968981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.969063 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.969138 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.970487 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.970494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.970503 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.970523 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.970561 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.970578 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.970702 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.970951 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.971008 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:44:41.972124 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.967952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33917 (local address 127.13.5.65:35205)
0928 15:44:41.968572 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.968593 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.968802 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.969397 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.969405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.969439 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.970234 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.970240 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.970257 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.970303 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.970803 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.970808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.970814 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.971512 (+ 698us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.971575 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":502,"threads_started":1}
I20260928 15:44:41.972532 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.966953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39101 (local address 127.13.5.65:60015)
0928 15:44:41.967755 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.967776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.968014 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.968567 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.968574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.968605 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.969450 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.969456 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.969481 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.969567 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.970210 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.970218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.970226 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.970392 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.970465 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":429,"threads_started":1}
I20260928 15:44:41.973111 18255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ce17558a0f42b2a497c89d236b477a" candidate_uuid: "923052f6d29745448c5e8f697ebea001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d83db656324e98a1e4407ab7149d4d" is_pre_election: true
I20260928 15:44:41.973625 18255 raft_consensus.cc:2454] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923052f6d29745448c5e8f697ebea001 in term 0.
I20260928 15:44:41.973694 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ce17558a0f42b2a497c89d236b477a" candidate_uuid: "923052f6d29745448c5e8f697ebea001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983cba850c0949fa83530a1f567b8897" is_pre_election: true
I20260928 15:44:41.974323 18385 raft_consensus.cc:2454] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923052f6d29745448c5e8f697ebea001 in term 0.
I20260928 15:44:41.974378 18059 leader_election.cc:302] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d83db656324e98a1e4407ab7149d4d, 923052f6d29745448c5e8f697ebea001; no voters:
I20260928 15:44:41.974499 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.968794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:57321 (local address 127.13.5.67:39101)
0928 15:44:41.971389 (+ 2595us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.971398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.971420 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:44:41.971477 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.971484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.971578 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.971676 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.973633 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.973638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.973645 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.973666 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.973707 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.973724 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.973859 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.974074 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.974130 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2497,"thread_start_us":1316,"threads_started":1}
I20260928 15:44:41.974824 18455 raft_consensus.cc:2781] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:41.975013 18455 raft_consensus.cc:482] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:41.975085 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.971924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:59053 (local address 127.13.5.65:39463)
0928 15:44:41.972101 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:41.972106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:41.972127 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:44:41.972174 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:41.972178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:41.972226 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:41.972322 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:41.974263 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:41.974268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:41.974274 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:41.974293 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:41.974325 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:41.974339 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:41.974426 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:41.974682 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 15:44:41.974742 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:44:41.975126 18455 raft_consensus.cc:3037] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:41.975243 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.968525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39101 (local address 127.13.5.66:57321)
0928 15:44:41.971044 (+ 2519us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.971060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.971145 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.972322 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.972332 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.972360 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.973414 (+ 1054us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.973419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.973437 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.973491 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.973922 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.973926 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.973931 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.974713 (+ 782us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.974762 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2362,"thread_start_us":2198,"threads_started":1}
I20260928 15:44:41.975421 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:41.971175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.13.5.66:59053)
0928 15:44:41.971901 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:41.971920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:41.972022 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:41.972473 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:41.972478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:41.972501 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:41.974057 (+ 1556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:44:41.974060 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:41.974076 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:44:41.974118 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:41.974539 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:41.974544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:41.974547 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 15:44:41.974916 (+ 369us) client_negotiation.cc:240] Negotiation successful
0928 15:44:41.974958 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":414,"threads_started":1}
I20260928 15:44:41.976257 18124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ce17558a0f42b2a497c89d236b477a" candidate_uuid: "40d83db656324e98a1e4407ab7149d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923052f6d29745448c5e8f697ebea001" is_pre_election: true
I20260928 15:44:41.977099 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ce17558a0f42b2a497c89d236b477a" candidate_uuid: "40d83db656324e98a1e4407ab7149d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983cba850c0949fa83530a1f567b8897" is_pre_election: true
I20260928 15:44:41.977310 18385 raft_consensus.cc:2454] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d83db656324e98a1e4407ab7149d4d in term 0.
I20260928 15:44:41.977752 18190 leader_election.cc:302] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d83db656324e98a1e4407ab7149d4d, 983cba850c0949fa83530a1f567b8897; no voters:
I20260928 15:44:41.978178 18456 raft_consensus.cc:2781] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:41.978322 18456 raft_consensus.cc:482] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:41.978223 18455 raft_consensus.cc:504] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.978513 18456 raft_consensus.cc:3037] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:41.978722 18124 raft_consensus.cc:2379] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40d83db656324e98a1e4407ab7149d4d in current term 1: Already voted for candidate 923052f6d29745448c5e8f697ebea001 in this term.
I20260928 15:44:41.979410 18385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ce17558a0f42b2a497c89d236b477a" candidate_uuid: "923052f6d29745448c5e8f697ebea001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983cba850c0949fa83530a1f567b8897"
I20260928 15:44:41.979629 18385 raft_consensus.cc:3037] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:41.980130 18455 leader_election.cc:288] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [CANDIDATE]: Term 1 election: Requested vote from peers 983cba850c0949fa83530a1f567b8897 (127.13.5.67:39101), 40d83db656324e98a1e4407ab7149d4d (127.13.5.66:33917)
I20260928 15:44:41.981160 18456 raft_consensus.cc:504] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.981709 18456 leader_election.cc:288] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [CANDIDATE]: Term 1 election: Requested vote from peers 983cba850c0949fa83530a1f567b8897 (127.13.5.67:39101), 923052f6d29745448c5e8f697ebea001 (127.13.5.65:39463)
I20260928 15:44:41.982172 18124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ce17558a0f42b2a497c89d236b477a" candidate_uuid: "40d83db656324e98a1e4407ab7149d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923052f6d29745448c5e8f697ebea001"
I20260928 15:44:41.982203 18384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ce17558a0f42b2a497c89d236b477a" candidate_uuid: "40d83db656324e98a1e4407ab7149d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983cba850c0949fa83530a1f567b8897"
I20260928 15:44:41.982399 18124 raft_consensus.cc:2379] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40d83db656324e98a1e4407ab7149d4d in current term 1: Already voted for candidate 923052f6d29745448c5e8f697ebea001 in this term.
I20260928 15:44:41.982656 18385 raft_consensus.cc:2454] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923052f6d29745448c5e8f697ebea001 in term 1.
I20260928 15:44:41.982975 18190 leader_election.cc:302] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40d83db656324e98a1e4407ab7149d4d; no voters: 923052f6d29745448c5e8f697ebea001, 983cba850c0949fa83530a1f567b8897
I20260928 15:44:41.983119 18059 leader_election.cc:302] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923052f6d29745448c5e8f697ebea001, 983cba850c0949fa83530a1f567b8897; no voters:
I20260928 15:44:41.983314 18456 raft_consensus.cc:2726] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 15:44:41.983364 18255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ce17558a0f42b2a497c89d236b477a" candidate_uuid: "923052f6d29745448c5e8f697ebea001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d83db656324e98a1e4407ab7149d4d"
I20260928 15:44:41.983608 18255 raft_consensus.cc:2379] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 923052f6d29745448c5e8f697ebea001 in current term 1: Already voted for candidate 40d83db656324e98a1e4407ab7149d4d in this term.
I20260928 15:44:41.983593 18455 raft_consensus.cc:2781] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:41.984155 18455 raft_consensus.cc:686] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 1 LEADER]: Becoming Leader. State: Replica: 923052f6d29745448c5e8f697ebea001, State: Running, Role: LEADER
I20260928 15:44:41.984707 18455 consensus_queue.cc:237] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } }
I20260928 15:44:41.988915 17981 catalog_manager.cc:5495] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 923052f6d29745448c5e8f697ebea001 (127.13.5.65). New cstate: current_term: 1 leader_uuid: "923052f6d29745448c5e8f697ebea001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923052f6d29745448c5e8f697ebea001" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39463 } health_report { overall_health: HEALTHY } } }
W20260928 15:44:42.013362 18431 tablet.cc:2369] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:44:42.032039 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.029359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.13.5.65:39463)
0928 15:44:42.029588 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:42.029595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:42.029681 (+ 86us) server_negotiation.cc:407] Connection header received
0928 15:44:42.029966 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:42.029974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:42.030115 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:42.030236 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:42.031310 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:42.031316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:42.031321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:42.031341 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:42.031384 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:42.031400 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:42.031528 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:42.031612 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 15:44:42.031661 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:44:42.040611 18255 raft_consensus.cc:1261] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 1 FOLLOWER]: Refusing update from remote peer 923052f6d29745448c5e8f697ebea001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:42.040601 18385 raft_consensus.cc:1261] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 1 FOLLOWER]: Refusing update from remote peer 923052f6d29745448c5e8f697ebea001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:42.041464 18455 consensus_queue.cc:1035] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40d83db656324e98a1e4407ab7149d4d" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:42.041826 18462 consensus_queue.cc:1035] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "983cba850c0949fa83530a1f567b8897" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:42.054646 18466 mvcc.cc:205] Tried to move back new op lower bound from 7334339715227746304 to 7334339715013697536. Current Snapshot: MvccSnapshot[applied={T|T < 7334339715227746304}]
I20260928 15:44:42.068926 17981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:42.069289 17981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:42.073217 17981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c73c6073d24d4c0dbc7fe36824b5caa2: Sending DeleteTablet for 3 replicas of tablet 30ce17558a0f42b2a497c89d236b477a
I20260928 15:44:42.074685 18235 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ce17558a0f42b2a497c89d236b477a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:42 UTC) from {username='slave'} at 127.0.0.1:50034
I20260928 15:44:42.074724 18365 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ce17558a0f42b2a497c89d236b477a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:42 UTC) from {username='slave'} at 127.0.0.1:51262
I20260928 15:44:42.075405 18104 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ce17558a0f42b2a497c89d236b477a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:42 UTC) from {username='slave'} at 127.0.0.1:38028
I20260928 15:44:42.076457 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 18042
I20260928 15:44:42.080412 18474 tablet_replica.cc:330] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: stopping tablet replica
I20260928 15:44:42.080947 18474 raft_consensus.cc:2229] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:42.081583 18474 raft_consensus.cc:2258] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:42.091500 18475 tablet_replica.cc:330] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: stopping tablet replica
I20260928 15:44:42.091995 18475 raft_consensus.cc:2229] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:42.092561 18475 raft_consensus.cc:2258] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:42.092636 18476 tablet_replica.cc:330] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001: stopping tablet replica
I20260928 15:44:42.093080 18476 raft_consensus.cc:2229] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:44:42.093670 18476 raft_consensus.cc:2258] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:42.094074 18474 ts_tablet_manager.cc:1905] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:42.094589 18475 ts_tablet_manager.cc:1905] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:42.095314 18476 ts_tablet_manager.cc:1905] T 30ce17558a0f42b2a497c89d236b477a P 923052f6d29745448c5e8f697ebea001: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:42.096974 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.078147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.13.5.65:39463)
0928 15:44:42.078379 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:42.078388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:42.089106 (+ 10718us) server_negotiation.cc:407] Connection header received
0928 15:44:42.089595 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:42.089605 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:42.089694 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:42.089782 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:42.095788 (+ 6006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:42.095793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:42.095798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:42.095821 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:42.095887 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:42.095907 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:42.096054 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:42.096617 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 15:44:42.096683 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:44:42.097704 18164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:42.103536 18474 ts_tablet_manager.cc:1918] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:42.103772 18474 log.cc:1196] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/30ce17558a0f42b2a497c89d236b477a
I20260928 15:44:42.104277 18474 ts_tablet_manager.cc:1939] T 30ce17558a0f42b2a497c89d236b477a P 983cba850c0949fa83530a1f567b8897: Deleting consensus metadata
I20260928 15:44:42.104245 18475 ts_tablet_manager.cc:1918] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:42.104435 18475 log.cc:1196] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/30ce17558a0f42b2a497c89d236b477a
I20260928 15:44:42.105082 18475 ts_tablet_manager.cc:1939] T 30ce17558a0f42b2a497c89d236b477a P 40d83db656324e98a1e4407ab7149d4d: Deleting consensus metadata
I20260928 15:44:42.106043 17969 catalog_manager.cc:4841] TS 983cba850c0949fa83530a1f567b8897 (127.13.5.67:39101): tablet 30ce17558a0f42b2a497c89d236b477a (table test-table [id=ed176b7b8ae04da19ca5614404b69f06]) successfully deleted
I20260928 15:44:42.107276 17969 catalog_manager.cc:4841] TS 40d83db656324e98a1e4407ab7149d4d (127.13.5.66:33917): tablet 30ce17558a0f42b2a497c89d236b477a (table test-table [id=ed176b7b8ae04da19ca5614404b69f06]) successfully deleted
I20260928 15:44:42.202075 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 18042
W20260928 15:44:42.221254 17969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.5.65:39463 (error 108)
I20260928 15:44:42.221359 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 18173
I20260928 15:44:42.223129 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.222689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.0.0.1:38056)
0928 15:44:42.222860 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:42.222975 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:44:42.223371 17969 catalog_manager.cc:4584] TS 923052f6d29745448c5e8f697ebea001 (127.13.5.65:39463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30ce17558a0f42b2a497c89d236b477a: Network error: Client connection negotiation failed: client connection to 127.13.5.65:39463: connect: Connection refused (error 111)
I20260928 15:44:42.224521 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.222183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.13.5.66:33917)
0928 15:44:42.222368 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:42.222376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:42.222474 (+ 98us) server_negotiation.cc:407] Connection header received
0928 15:44:42.222692 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:42.222697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:42.222781 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:42.222894 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:42.223844 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:42.223856 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:42.223859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:42.223913 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:42.223950 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:42.223971 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:42.224103 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:42.224192 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 15:44:42.224253 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:44:42.225198 18295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:42.277055 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.276492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.0.0.1:38064)
0928 15:44:42.276737 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:42.276857 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:44:42.278403 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.277957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.0.0.1:38076)
0928 15:44:42.278137 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:42.278226 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:44:42.317811 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 18173
I20260928 15:44:42.336417 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 18303
I20260928 15:44:42.339038 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.338637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.0.0.1:38088)
0928 15:44:42.338806 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:42.338893 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 15:44:42.340123 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.339690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.0.0.1:38098)
0928 15:44:42.339843 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:42.339978 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:44:42.340463 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.337244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.13.5.67:39101)
0928 15:44:42.337460 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:42.337467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:42.337548 (+ 81us) server_negotiation.cc:407] Connection header received
0928 15:44:42.337703 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:42.337709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:42.337810 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:42.337933 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:42.339620 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:42.339627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:42.339632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:42.339652 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:42.339691 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:42.339717 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:42.339831 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:42.340108 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 15:44:42.340171 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:44:42.341126 18425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:42.421058 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.420503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.0.0.1:38112)
0928 15:44:42.420768 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:42.420891 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 15:44:42.422417 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:42.421953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39463 (local address 127.0.0.1:38122)
0928 15:44:42.422178 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:42.422271 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:39463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 15:44:42.431129 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 18303
I20260928 15:44:42.449121 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 17952
I20260928 15:44:42.450337 18013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:42.557694 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 17952
2026-09-28T15:44:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2713 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 15:44:42.600023 13333 mini_oidc.cc:152] Starting JWKS server
I20260928 15:44:42.602705 13333 webserver.cc:460] Webserver started at https://127.0.0.1:44465/ using document root <none> and password file <none>
I20260928 15:44:42.603048 13333 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:44:42.603724 13333 webserver.cc:460] Webserver started at http://127.0.0.1:36971/ using document root <none> and password file <none>
2026-09-28T15:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:42Z Disabled control of system clock
I20260928 15:44:42.631410 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43735
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43735
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44465/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/testchainca.cert with env {}
W20260928 15:44:42.960342 18488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:42.960785 18488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:42.960934 18488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:42.968228 18488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:42.968348 18488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:42.968465 18488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:42.968575 18488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:42.968662 18488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:44:42.968775 18488 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:44:42.968855 18488 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44465/jwks.json
I20260928 15:44:42.979027 18488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44465/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:42.980516 18488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:42.982275 18488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:42.994599 18493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:42.995029 18496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:42.995093 18494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:42.995607 18488 server_base.cc:1034] running on GCE node
I20260928 15:44:42.996567 18488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:42.998138 18488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:42.999361 18488 hybrid_clock.cc:648] HybridClock initialized: now 1790610282999288 us; error 79 us; skew 500 ppm
I20260928 15:44:42.999845 18488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:44:43.002321 18488 webserver.cc:460] Webserver started at http://127.13.5.126:45685/ using document root <none> and password file <none>
I20260928 15:44:43.003072 18488 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:43.003237 18488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:43.003566 18488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:43.006875 18488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "c6d5ed01648d4ea7944b700feec922f8"
format_stamp: "Formatted at 2026-09-28 15:44:43 on dist-test-slave-j968"
server_key: "0596b017189b6b90b1d10dde85bdd31e"
server_key_iv: "f1f8c58ae0f5e5cb524a305bb3ebd187"
server_key_version: "encryptionkey@0"
I20260928 15:44:43.007717 18488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "c6d5ed01648d4ea7944b700feec922f8"
format_stamp: "Formatted at 2026-09-28 15:44:43 on dist-test-slave-j968"
server_key: "0596b017189b6b90b1d10dde85bdd31e"
server_key_iv: "f1f8c58ae0f5e5cb524a305bb3ebd187"
server_key_version: "encryptionkey@0"
I20260928 15:44:43.013033 18488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:44:43.016682 18502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:43.018106 18488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:43.018312 18488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "c6d5ed01648d4ea7944b700feec922f8"
format_stamp: "Formatted at 2026-09-28 15:44:43 on dist-test-slave-j968"
server_key: "0596b017189b6b90b1d10dde85bdd31e"
server_key_iv: "f1f8c58ae0f5e5cb524a305bb3ebd187"
server_key_version: "encryptionkey@0"
I20260928 15:44:43.018603 18488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:43.041024 18488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:43.041978 18488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:43.042282 18488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:43.061537 18488 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43735
I20260928 15:44:43.061559 18553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43735 every 8 connection(s)
I20260928 15:44:43.062947 18488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:43.067063 18554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:43.069919 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 18488
I20260928 15:44:43.070158 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:43.070977 13333 external_mini_cluster.cc:1428] Setting key 2fbc9a3d32b141ba9bfb27f4af97f934
I20260928 15:44:43.079913 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8: Bootstrap starting.
I20260928 15:44:43.083918 18554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:43.085101 18554 log.cc:824] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:43.085623 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:43.073109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.13.5.126:43735)
0928 15:44:43.074148 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:43.074159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:43.074201 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:44:43.075953 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:43.075989 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:43.076503 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:43.076849 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:43.077531 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.078849 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.080724 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.081526 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.081680 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:43.083044 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:43.083052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:43.083065 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:43.083644 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:43.083731 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:43.083752 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:43.083842 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:43.084196 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 15:44:43.084591 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":414,"threads_started":1}
I20260928 15:44:43.088186 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8: No bootstrap required, opened a new log
I20260928 15:44:43.089715 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:44:43.096102 18554 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d5ed01648d4ea7944b700feec922f8" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43735 } }
I20260928 15:44:43.096499 18554 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:43.096624 18554 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d5ed01648d4ea7944b700feec922f8, State: Initialized, Role: FOLLOWER
I20260928 15:44:43.097186 18554 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d5ed01648d4ea7944b700feec922f8" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43735 } }
I20260928 15:44:43.097386 18554 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:43.097517 18554 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:43.097692 18554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:43.099843 18554 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d5ed01648d4ea7944b700feec922f8" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43735 } }
I20260928 15:44:43.100473 18554 leader_election.cc:302] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d5ed01648d4ea7944b700feec922f8; no voters:
I20260928 15:44:43.101035 18554 leader_election.cc:288] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:43.101208 18559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:43.101961 18559 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [term 1 LEADER]: Becoming Leader. State: Replica: c6d5ed01648d4ea7944b700feec922f8, State: Running, Role: LEADER
I20260928 15:44:43.102591 18559 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d5ed01648d4ea7944b700feec922f8" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43735 } }
I20260928 15:44:43.103231 18554 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:43.105957 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d5ed01648d4ea7944b700feec922f8. Latest consensus state: current_term: 1 leader_uuid: "c6d5ed01648d4ea7944b700feec922f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d5ed01648d4ea7944b700feec922f8" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43735 } } }
I20260928 15:44:43.106427 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:43.106626 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6d5ed01648d4ea7944b700feec922f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d5ed01648d4ea7944b700feec922f8" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43735 } } }
I20260928 15:44:43.107038 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:43.108507 18568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:43.114442 18568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:43.124989 18568 catalog_manager.cc:1347] Generated new cluster ID: 5e2213ed8b3a42b0972cae9d94ca4efc
I20260928 15:44:43.125149 18568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:43.146555 18568 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:43.148789 18568 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:43.163956 18568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d5ed01648d4ea7944b700feec922f8: Generated new TSK 0
I20260928 15:44:43.164952 18568 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 15:44:43.215971 18584 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 15:44:43.216158 18584 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:44:43.216974 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:43.187542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.13.5.126:43735)
0928 15:44:43.187787 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:43.187795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:43.188236 (+ 441us) server_negotiation.cc:407] Connection header received
0928 15:44:43.188659 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:43.188665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:43.188767 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:43.188923 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:44:43.189559 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.190349 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.192547 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.192978 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.193072 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:43.193740 (+ 668us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:44:43.216402 (+ 22662us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 15:44:43.216748 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":100}
W20260928 15:44:43.217123 18557 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 15:44:43.218963 18583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:44:43.187350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43735 (local address 127.0.0.1:59862)
0928 15:44:43.188108 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:43.188131 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:43.188331 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:43.188982 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:43.188988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:43.189002 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:44:43.189376 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:43.189389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.190507 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.190514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:43.192389 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:43.192406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.193161 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.193168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:43.193377 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:43.193460 (+ 83us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:44:43.216721 (+ 23261us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 15:44:43.217101 (+ 380us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.5.126:43735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":916,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.5.126:43735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 15:44:43.221181 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 18488
I20260928 15:44:43.222524 18549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:43.317586 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 18488
2026-09-28T15:44:43Z chronyd exiting
I20260928 15:44:43.368817 13333 test_util.cc:175] -----------------------------------------------
I20260928 15:44:43.368963 13333 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610238672534-13333-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (773 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 15:44:43.372599 13333 mini_oidc.cc:152] Starting JWKS server
I20260928 15:44:43.374440 13333 webserver.cc:460] Webserver started at https://127.0.0.1:36287/ using document root <none> and password file <none>
I20260928 15:44:43.374636 13333 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:44:43.375205 13333 webserver.cc:460] Webserver started at http://127.0.0.1:37469/ using document root <none> and password file <none>
2026-09-28T15:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:43Z Disabled control of system clock
I20260928 15:44:43.400691 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:39563
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:39563
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36287/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/testchainca.cert with env {}
W20260928 15:44:43.729451 18591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:43.729769 18591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:43.729965 18591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:43.737524 18591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:43.737636 18591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:43.737733 18591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:43.737821 18591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:43.737888 18591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:44:43.737982 18591 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:44:43.738057 18591 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36287/jwks.json
I20260928 15:44:43.748291 18591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:39563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36287/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:43.749706 18591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:43.751078 18591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:43.763710 18596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:43.763777 18597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:43.763765 18599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:43.764667 18591 server_base.cc:1034] running on GCE node
I20260928 15:44:43.765542 18591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:43.767082 18591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:43.768332 18591 hybrid_clock.cc:648] HybridClock initialized: now 1790610283768273 us; error 42 us; skew 500 ppm
I20260928 15:44:43.768738 18591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:44:43.771252 18591 webserver.cc:460] Webserver started at http://127.13.5.126:38023/ using document root <none> and password file <none>
I20260928 15:44:43.772119 18591 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:43.772262 18591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:43.772614 18591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:43.775735 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "d3269a25f2ac4580b753544092e10832"
format_stamp: "Formatted at 2026-09-28 15:44:43 on dist-test-slave-j968"
server_key: "3ae63800a45c0daa4ffec92a192c5689"
server_key_iv: "1d5ba9957ee94bd6864129b15243747b"
server_key_version: "encryptionkey@0"
I20260928 15:44:43.776712 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "d3269a25f2ac4580b753544092e10832"
format_stamp: "Formatted at 2026-09-28 15:44:43 on dist-test-slave-j968"
server_key: "3ae63800a45c0daa4ffec92a192c5689"
server_key_iv: "1d5ba9957ee94bd6864129b15243747b"
server_key_version: "encryptionkey@0"
I20260928 15:44:43.782199 18591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:44:43.785868 18605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:43.787266 18591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:44:43.787461 18591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "d3269a25f2ac4580b753544092e10832"
format_stamp: "Formatted at 2026-09-28 15:44:43 on dist-test-slave-j968"
server_key: "3ae63800a45c0daa4ffec92a192c5689"
server_key_iv: "1d5ba9957ee94bd6864129b15243747b"
server_key_version: "encryptionkey@0"
I20260928 15:44:43.787694 18591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:43.800977 18591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:43.801908 18591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:43.802215 18591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:43.821149 18591 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:39563
I20260928 15:44:43.821168 18656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:39563 every 8 connection(s)
I20260928 15:44:43.822633 18591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:43.826792 18657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:43.828678 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 18591
I20260928 15:44:43.828877 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:43.829591 13333 external_mini_cluster.cc:1428] Setting key 10cc122a8e76278065d4e30033067ca3
I20260928 15:44:43.838521 18657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832: Bootstrap starting.
I20260928 15:44:43.841936 18657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:43.843012 18657 log.cc:824] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:43.843320 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:43.831627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.13.5.126:39563)
0928 15:44:43.832704 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:43.832726 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:43.832769 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:44:43.834409 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:43.834431 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:43.834906 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:43.835312 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:43.835965 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.837299 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.838641 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.839367 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.839585 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:43.840869 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:43.840876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:43.840891 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:43.841467 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:43.841565 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:43.841586 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:43.841724 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:43.841959 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 15:44:43.842327 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":392,"threads_started":1}
I20260928 15:44:43.846158 18657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832: No bootstrap required, opened a new log
I20260928 15:44:43.847260 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:44:43.854190 18657 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3269a25f2ac4580b753544092e10832" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39563 } }
I20260928 15:44:43.854592 18657 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:43.854699 18657 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3269a25f2ac4580b753544092e10832, State: Initialized, Role: FOLLOWER
I20260928 15:44:43.855273 18657 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3269a25f2ac4580b753544092e10832" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39563 } }
I20260928 15:44:43.855531 18657 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:43.855679 18657 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:43.855851 18657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:43.858316 18657 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3269a25f2ac4580b753544092e10832" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39563 } }
I20260928 15:44:43.858836 18657 leader_election.cc:302] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3269a25f2ac4580b753544092e10832; no voters:
I20260928 15:44:43.859418 18657 leader_election.cc:288] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:43.859660 18662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:43.860419 18662 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [term 1 LEADER]: Becoming Leader. State: Replica: d3269a25f2ac4580b753544092e10832, State: Running, Role: LEADER
I20260928 15:44:43.860949 18662 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3269a25f2ac4580b753544092e10832" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39563 } }
I20260928 15:44:43.861660 18657 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:43.864138 18663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3269a25f2ac4580b753544092e10832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3269a25f2ac4580b753544092e10832" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39563 } } }
I20260928 15:44:43.864122 18664 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3269a25f2ac4580b753544092e10832. Latest consensus state: current_term: 1 leader_uuid: "d3269a25f2ac4580b753544092e10832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3269a25f2ac4580b753544092e10832" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39563 } } }
I20260928 15:44:43.864557 18663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:43.864621 18664 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:43.866153 18668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:43.871496 18668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:43.880903 18668 catalog_manager.cc:1347] Generated new cluster ID: 12f2455fb8f441e1ad28ab6bc0e72c8c
I20260928 15:44:43.881027 18668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:43.894032 18668 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:43.895846 18668 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:43.908922 18668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3269a25f2ac4580b753544092e10832: Generated new TSK 0
I20260928 15:44:43.910156 18668 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 15:44:43.944698 18687 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 15:44:43.944836 18687 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:44:43.946087 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:43.925823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.13.5.126:39563)
0928 15:44:43.926038 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:43.926045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:43.926500 (+ 455us) server_negotiation.cc:407] Connection header received
0928 15:44:43.926731 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:43.926736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:43.926831 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:43.927022 (+ 191us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:44:43.927471 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.928615 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.929965 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.930658 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.930786 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:43.931301 (+ 515us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:44:43.945390 (+ 14089us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 15:44:43.945769 (+ 379us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 15:44:43.946185 18686 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:44:43.925713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39563 (local address 127.0.0.1:51098)
0928 15:44:43.926391 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:43.926411 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:43.926539 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:43.927044 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:43.927048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:43.927057 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:44:43.927309 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:43.927316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.928759 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.928763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:43.929819 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:43.929830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:43.930839 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:43.930843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:43.930973 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:43.931025 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:44:43.945626 (+ 14601us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 15:44:43.945826 (+ 200us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.5.126:39563: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":391,"threads_started":1}
W20260928 15:44:43.946210 18660 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36287/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 15:44:43.947956 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 18591
I20260928 15:44:43.949074 18652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:44.042814 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 18591
2026-09-28T15:44:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (729 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 15:44:44.102315 13333 mini_oidc.cc:152] Starting JWKS server
I20260928 15:44:44.104172 13333 webserver.cc:460] Webserver started at https://127.0.0.1:43581/ using document root <none> and password file <none>
I20260928 15:44:44.104373 13333 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:44:44.104928 13333 webserver.cc:460] Webserver started at http://127.0.0.1:33685/ using document root <none> and password file <none>
2026-09-28T15:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:44Z Disabled control of system clock
I20260928 15:44:44.132490 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:37319
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:37319
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43581/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 15:44:44.468365 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:44.468714 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:44.468864 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:44.476233 18694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:44.476375 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:44.476465 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:44.476581 18694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:44.476660 18694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:44:44.476745 18694 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:44:44.476845 18694 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43581/jwks.json
I20260928 15:44:44.486174 18694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:37319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43581/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:44.487527 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:44.489444 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:44.502494 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:44.502502 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:44.502544 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:44.503224 18694 server_base.cc:1034] running on GCE node
I20260928 15:44:44.504132 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:44.505573 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:44.506819 18694 hybrid_clock.cc:648] HybridClock initialized: now 1790610284506743 us; error 76 us; skew 500 ppm
I20260928 15:44:44.507211 18694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:44:44.509539 18694 webserver.cc:460] Webserver started at http://127.13.5.126:42179/ using document root <none> and password file <none>
I20260928 15:44:44.510267 18694 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:44.510414 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:44.510744 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:44.513689 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "3ae2862a236f429586c21e4fe0877329"
format_stamp: "Formatted at 2026-09-28 15:44:44 on dist-test-slave-j968"
server_key: "0a45d528ae0b7bf5252feaed115a8597"
server_key_iv: "d637b3ad2b0d15194629f68f863bb8cf"
server_key_version: "encryptionkey@0"
I20260928 15:44:44.514519 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "3ae2862a236f429586c21e4fe0877329"
format_stamp: "Formatted at 2026-09-28 15:44:44 on dist-test-slave-j968"
server_key: "0a45d528ae0b7bf5252feaed115a8597"
server_key_iv: "d637b3ad2b0d15194629f68f863bb8cf"
server_key_version: "encryptionkey@0"
I20260928 15:44:44.519548 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:44:44.523095 18708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:44.524557 18694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:44:44.524746 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "3ae2862a236f429586c21e4fe0877329"
format_stamp: "Formatted at 2026-09-28 15:44:44 on dist-test-slave-j968"
server_key: "0a45d528ae0b7bf5252feaed115a8597"
server_key_iv: "d637b3ad2b0d15194629f68f863bb8cf"
server_key_version: "encryptionkey@0"
I20260928 15:44:44.524967 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:44.548897 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:44.549757 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:44.550061 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:44.569015 18694 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:37319
I20260928 15:44:44.569034 18759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:37319 every 8 connection(s)
I20260928 15:44:44.570434 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:44.574195 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 18694
I20260928 15:44:44.574419 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:44.574713 18760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:44.575251 13333 external_mini_cluster.cc:1428] Setting key 206fff02842151df0f05c0c73b70afbd
I20260928 15:44:44.589725 18760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329: Bootstrap starting.
I20260928 15:44:44.589922 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:44.577382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.13.5.126:37319)
0928 15:44:44.578706 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:44.578718 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:44.578763 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:44:44.580451 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:44.580479 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:44.580959 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:44.581329 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:44.581961 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:44.583347 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:44.585252 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:44.585971 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:44.586116 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:44.587395 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:44.587402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:44.587424 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:44.588094 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:44.588172 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:44.588193 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:44.588286 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:44.588549 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 15:44:44.588904 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":427,"threads_started":1}
I20260928 15:44:44.593290 18760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:44.594014 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:44:44.594542 18760 log.cc:824] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:44.597640 18760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329: No bootstrap required, opened a new log
I20260928 15:44:44.605382 18760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae2862a236f429586c21e4fe0877329" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37319 } }
I20260928 15:44:44.605906 18760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:44.606017 18760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae2862a236f429586c21e4fe0877329, State: Initialized, Role: FOLLOWER
I20260928 15:44:44.606691 18760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae2862a236f429586c21e4fe0877329" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37319 } }
I20260928 15:44:44.606900 18760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:44.606998 18760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:44.607122 18760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:44.609256 18760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae2862a236f429586c21e4fe0877329" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37319 } }
I20260928 15:44:44.609787 18760 leader_election.cc:302] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ae2862a236f429586c21e4fe0877329; no voters:
I20260928 15:44:44.610318 18760 leader_election.cc:288] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:44.610499 18765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:44.611219 18765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [term 1 LEADER]: Becoming Leader. State: Replica: 3ae2862a236f429586c21e4fe0877329, State: Running, Role: LEADER
I20260928 15:44:44.611931 18765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae2862a236f429586c21e4fe0877329" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37319 } }
I20260928 15:44:44.612850 18760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:44.615187 18767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ae2862a236f429586c21e4fe0877329. Latest consensus state: current_term: 1 leader_uuid: "3ae2862a236f429586c21e4fe0877329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae2862a236f429586c21e4fe0877329" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37319 } } }
I20260928 15:44:44.615695 18767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:44.615199 18766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ae2862a236f429586c21e4fe0877329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae2862a236f429586c21e4fe0877329" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37319 } } }
I20260928 15:44:44.617120 18766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:44.617210 18771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:44.622804 18771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:44.632576 18771 catalog_manager.cc:1347] Generated new cluster ID: e3da8220c6eb4bf89e1257f9072aee6f
I20260928 15:44:44.632697 18771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:44.646023 18771 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:44.648031 18771 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:44.661429 18771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ae2862a236f429586c21e4fe0877329: Generated new TSK 0
I20260928 15:44:44.662580 18771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:44:44.691069 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:44.681817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.13.5.126:37319)
0928 15:44:44.682100 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:44.682108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:44.682467 (+ 359us) server_negotiation.cc:407] Connection header received
0928 15:44:44.682711 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:44.682716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:44.682797 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:44.682962 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:44:44.683568 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:44.684400 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:44.686487 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:44.686965 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:44.687071 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:44.687738 (+ 667us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:44:44.690391 (+ 2653us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 15:44:44.690750 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57658: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 15:44:44.691442 18763 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57658: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 15:44:44.693785 18789 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:44:44.681693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37319 (local address 127.0.0.1:57658)
0928 15:44:44.682330 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:44.682353 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:44.682524 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:44.683014 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:44.683022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:44.683036 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:44:44.683376 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:44.683388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:44.684545 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:44.684551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:44.686324 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:44.686341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:44.687163 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:44.687170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:44.687379 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:44.687428 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:44:44.691737 (+ 4309us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 15:44:44.693367 (+ 1630us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.5.126:37319: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43581/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":1219,"thread_start_us":345,"threads_started":1}
I20260928 15:44:44.696034 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 18694
I20260928 15:44:44.697774 18755 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 15:44:44.704463 18790 webserver.cc:533] Webserver: sslize failed:
E20260928 15:44:44.704548 18790 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:44:44.792001 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 18694
2026-09-28T15:44:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (746 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T15:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:44Z Disabled control of system clock
I20260928 15:44:44.876996 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:37707
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:37707
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/testchainca.cert with env {}
W20260928 15:44:45.203193 18795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:45.203501 18795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:45.203660 18795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:45.210851 18795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:45.210958 18795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:45.211050 18795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:45.211159 18795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:45.211241 18795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:44:45.211328 18795 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:44:45.211378 18795 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 15:44:45.221771 18795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:37707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:37707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:45.223137 18795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:45.224629 18795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:45.237080 18803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:45.237031 18801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:45.237388 18800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:45.237974 18795 server_base.cc:1034] running on GCE node
I20260928 15:44:45.238765 18795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:45.240188 18795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:45.241427 18795 hybrid_clock.cc:648] HybridClock initialized: now 1790610285241383 us; error 53 us; skew 500 ppm
I20260928 15:44:45.241801 18795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:44:45.244204 18795 webserver.cc:460] Webserver started at http://127.13.5.126:44665/ using document root <none> and password file <none>
I20260928 15:44:45.244907 18795 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:45.245047 18795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:45.245357 18795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:45.248234 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "0a14c145a157473ab4bf860436ae7dec"
format_stamp: "Formatted at 2026-09-28 15:44:45 on dist-test-slave-j968"
server_key: "cb41db7fd0c329939942b794a472a8f8"
server_key_iv: "91810caec7badb2f115891c1422032c0"
server_key_version: "encryptionkey@0"
I20260928 15:44:45.249027 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "0a14c145a157473ab4bf860436ae7dec"
format_stamp: "Formatted at 2026-09-28 15:44:45 on dist-test-slave-j968"
server_key: "cb41db7fd0c329939942b794a472a8f8"
server_key_iv: "91810caec7badb2f115891c1422032c0"
server_key_version: "encryptionkey@0"
I20260928 15:44:45.254264 18795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 15:44:45.257793 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:45.259331 18795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:44:45.259521 18795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "0a14c145a157473ab4bf860436ae7dec"
format_stamp: "Formatted at 2026-09-28 15:44:45 on dist-test-slave-j968"
server_key: "cb41db7fd0c329939942b794a472a8f8"
server_key_iv: "91810caec7badb2f115891c1422032c0"
server_key_version: "encryptionkey@0"
I20260928 15:44:45.259742 18795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:45.273209 18795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:45.274047 18795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:45.274333 18795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:45.293656 18795 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:37707
I20260928 15:44:45.293659 18860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:37707 every 8 connection(s)
I20260928 15:44:45.295163 18795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:45.295719 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 18795
I20260928 15:44:45.295949 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:45.296897 13333 external_mini_cluster.cc:1428] Setting key e16bf155fae903b9b3689dbe8e5882d2
I20260928 15:44:45.299659 18861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:45.311076 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:45.298988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.13.5.126:37707)
0928 15:44:45.300117 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:45.300128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:45.300169 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:44:45.301768 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:45.301789 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:45.302250 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:45.302581 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:45.303218 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:45.304726 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:45.306562 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:45.307214 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:45.307344 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:45.308645 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:45.308652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:45.308666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:44:45.309245 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:45.309347 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:44:45.309371 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:45.309472 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:45.309705 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 15:44:45.310070 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":362,"threads_started":1}
I20260928 15:44:45.312088 18861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec: Bootstrap starting.
I20260928 15:44:45.315196 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:44:45.315655 18861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:45.316732 18861 log.cc:824] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:45.319797 18861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec: No bootstrap required, opened a new log
I20260928 15:44:45.327684 18861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a14c145a157473ab4bf860436ae7dec" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37707 } }
I20260928 15:44:45.328190 18861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:45.328305 18861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a14c145a157473ab4bf860436ae7dec, State: Initialized, Role: FOLLOWER
I20260928 15:44:45.328912 18861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a14c145a157473ab4bf860436ae7dec" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37707 } }
I20260928 15:44:45.329166 18861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:45.329310 18861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:45.329495 18861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:45.332000 18861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a14c145a157473ab4bf860436ae7dec" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37707 } }
I20260928 15:44:45.332520 18861 leader_election.cc:302] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a14c145a157473ab4bf860436ae7dec; no voters:
I20260928 15:44:45.333122 18861 leader_election.cc:288] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:45.333276 18866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:45.334015 18866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [term 1 LEADER]: Becoming Leader. State: Replica: 0a14c145a157473ab4bf860436ae7dec, State: Running, Role: LEADER
I20260928 15:44:45.334586 18866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a14c145a157473ab4bf860436ae7dec" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37707 } }
I20260928 15:44:45.335469 18861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:45.337832 18868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a14c145a157473ab4bf860436ae7dec. Latest consensus state: current_term: 1 leader_uuid: "0a14c145a157473ab4bf860436ae7dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a14c145a157473ab4bf860436ae7dec" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37707 } } }
I20260928 15:44:45.337919 18867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a14c145a157473ab4bf860436ae7dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a14c145a157473ab4bf860436ae7dec" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37707 } } }
I20260928 15:44:45.338372 18868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:45.338420 18867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:45.340041 18875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:45.345150 18875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:45.354967 18875 catalog_manager.cc:1347] Generated new cluster ID: 2f481ac91d9e466e8ec481580f46ae49
I20260928 15:44:45.355096 18875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:45.375708 18875 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:45.377111 18875 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:45.386837 18875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a14c145a157473ab4bf860436ae7dec: Generated new TSK 0
I20260928 15:44:45.387708 18875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:44:45.434687 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:45.405916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.13.5.126:37707)
0928 15:44:45.406122 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:45.406128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:45.406585 (+ 457us) server_negotiation.cc:407] Connection header received
0928 15:44:45.406803 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:45.406809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:45.406900 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:45.407090 (+ 190us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:44:45.407513 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:45.408669 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:45.410026 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:45.410739 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:45.410864 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:45.411382 (+ 518us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:44:45.433907 (+ 22525us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 15:44:45.434357 (+ 450us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47318: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 15:44:45.434818 18890 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:44:45.405830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37707 (local address 127.0.0.1:47318)
0928 15:44:45.406483 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:45.406499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:45.406630 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:45.407105 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:45.407109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:45.407127 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:44:45.407360 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:45.407367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:45.408823 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:45.408827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:45.409882 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:45.409893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:45.410915 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:45.410919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:45.411066 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:45.411098 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:44:45.434247 (+ 23149us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 15:44:45.434478 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.5.126:37707: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":362,"threads_started":1}
W20260928 15:44:45.434834 18864 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47318: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 15:44:45.436268 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 18795
I20260928 15:44:45.437745 18856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:45.533577 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 18795
2026-09-28T15:44:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (719 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (418 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1512 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:47 dist-test-slave-j968 krb5kdc[18906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:47 dist-test-slave-j968 krb5kdc[18906](info): set up 2 sockets
Sep 28 15:44:47 dist-test-slave-j968 krb5kdc[18906](info): commencing operation
krb5kdc: starting...
W20260928 15:44:48.559051 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:48 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:44:48.780872 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43573
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:49.111399 18919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:49.111708 18919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:49.111903 18919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:49.119099 18919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:49.119205 18919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:49.119297 18919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:49.119400 18919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:49.119493 18919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:49.129520 18919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:49.130980 18919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:49.132483 18919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:49.145305 18927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:49.145771 18925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:49.145375 18924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:49.146284 18919 server_base.cc:1034] running on GCE node
I20260928 15:44:49.147096 18919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:49.148650 18919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:49.149886 18919 hybrid_clock.cc:648] HybridClock initialized: now 1790610289149827 us; error 51 us; skew 500 ppm
I20260928 15:44:49.150280 18919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:49 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610289, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:49.153594 18919 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:44:49.155303 18919 webserver.cc:460] Webserver started at http://127.13.5.126:46199/ using document root <none> and password file <none>
I20260928 15:44:49.156090 18919 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:49.156237 18919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:49.156572 18919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:49.159554 18919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "e7bb1fa6700c4e77b15200f98fcf83a5"
format_stamp: "Formatted at 2026-09-28 15:44:49 on dist-test-slave-j968"
server_key: "1920d2eb01c37ad3bf87125fa99f4218"
server_key_iv: "c51f51071ef196e4c3da361113b83531"
server_key_version: "encryptionkey@0"
I20260928 15:44:49.160360 18919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "e7bb1fa6700c4e77b15200f98fcf83a5"
format_stamp: "Formatted at 2026-09-28 15:44:49 on dist-test-slave-j968"
server_key: "1920d2eb01c37ad3bf87125fa99f4218"
server_key_iv: "c51f51071ef196e4c3da361113b83531"
server_key_version: "encryptionkey@0"
I20260928 15:44:49.165560 18919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:44:49.169267 18934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:49.170737 18919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 15:44:49.170919 18919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "e7bb1fa6700c4e77b15200f98fcf83a5"
format_stamp: "Formatted at 2026-09-28 15:44:49 on dist-test-slave-j968"
server_key: "1920d2eb01c37ad3bf87125fa99f4218"
server_key_iv: "c51f51071ef196e4c3da361113b83531"
server_key_version: "encryptionkey@0"
I20260928 15:44:49.171133 18919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:49.198040 18919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:49.202782 18919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:49.203091 18919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:49.222333 18919 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43573
I20260928 15:44:49.222388 18985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43573 every 8 connection(s)
I20260928 15:44:49.223824 18919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:49.227931 18986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:49.233357 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 18919
I20260928 15:44:49.233563 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:49.234355 13333 external_mini_cluster.cc:1428] Setting key 330af8c12be950f995ad387583b56832
I20260928 15:44:49.239845 18986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5: Bootstrap starting.
I20260928 15:44:49.243916 18986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:49.245245 18986 log.cc:824] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:49.249013 18986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5: No bootstrap required, opened a new log
Sep 28 15:44:49 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:49.257000 18986 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb1fa6700c4e77b15200f98fcf83a5" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43573 } }
I20260928 15:44:49.257503 18986 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:49.257627 18986 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bb1fa6700c4e77b15200f98fcf83a5, State: Initialized, Role: FOLLOWER
I20260928 15:44:49.258298 18986 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb1fa6700c4e77b15200f98fcf83a5" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43573 } }
I20260928 15:44:49.258553 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:49.236557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.13.5.126:43573)
0928 15:44:49.237423 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:49.237435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:49.237487 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:44:49.239215 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:49.239239 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:49.239648 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:49.240129 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:49.241893 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:49.243259 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:49.245102 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:49.245987 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:49.246198 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:49.250740 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:49.250748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:49.250766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:49.250815 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:49.254154 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:49.254881 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:49.254888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:49.254894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:49.255071 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:49.255492 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:49.255501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:49.255503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:49.256139 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:49.256465 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:49.256888 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 15:44:49.257331 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":413,"threads_started":1}
I20260928 15:44:49.258574 18986 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:49.258806 18986 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:49.259008 18986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:49.261885 18986 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb1fa6700c4e77b15200f98fcf83a5" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43573 } }
I20260928 15:44:49.262326 18986 leader_election.cc:302] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7bb1fa6700c4e77b15200f98fcf83a5; no voters:
I20260928 15:44:49.262894 18986 leader_election.cc:288] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:49.263038 18991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:49.263721 18991 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [term 1 LEADER]: Becoming Leader. State: Replica: e7bb1fa6700c4e77b15200f98fcf83a5, State: Running, Role: LEADER
I20260928 15:44:49.264411 18991 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb1fa6700c4e77b15200f98fcf83a5" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43573 } }
I20260928 15:44:49.265183 18986 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:49.267421 18993 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7bb1fa6700c4e77b15200f98fcf83a5. Latest consensus state: current_term: 1 leader_uuid: "e7bb1fa6700c4e77b15200f98fcf83a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb1fa6700c4e77b15200f98fcf83a5" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43573 } } }
I20260928 15:44:49.267544 18992 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7bb1fa6700c4e77b15200f98fcf83a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb1fa6700c4e77b15200f98fcf83a5" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43573 } } }
I20260928 15:44:49.267896 18993 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:49.267962 18992 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:49.270633 18998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:49.277452 18998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:49.287603 18998 catalog_manager.cc:1347] Generated new cluster ID: efdf155e327540cbb3eab27cec149872
I20260928 15:44:49.287730 18998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:49.303735 18998 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:49.305140 18998 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:49.314388 18998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7bb1fa6700c4e77b15200f98fcf83a5: Generated new TSK 0
I20260928 15:44:49.315220 18998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:44:49.372115 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43573
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:49.708506 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:49.708786 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:49.708880 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:49.716122 19012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:49.716233 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:49.716343 19012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:49.726126 19012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:49.727418 19012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:49.728739 19012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:49.742043 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:49.742187 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:49.742091 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:49.743099 19012 server_base.cc:1034] running on GCE node
I20260928 15:44:49.743826 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:49.745296 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:49.746613 19012 hybrid_clock.cc:648] HybridClock initialized: now 1790610289746505 us; error 95 us; skew 500 ppm
I20260928 15:44:49.746974 19012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:49 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610289, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:49.750342 19012 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:44:49.752115 19012 webserver.cc:460] Webserver started at http://127.13.5.65:45783/ using document root <none> and password file <none>
I20260928 15:44:49.752844 19012 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:49.752992 19012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:49.753310 19012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:49.756378 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "c12372d6ffdb4819ae77869af17dc8df"
format_stamp: "Formatted at 2026-09-28 15:44:49 on dist-test-slave-j968"
server_key: "ad6a8973c9202acdcd71f7a482c14756"
server_key_iv: "67cdd04fab8e878eb9c57de4a2750d5b"
server_key_version: "encryptionkey@0"
I20260928 15:44:49.757184 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "c12372d6ffdb4819ae77869af17dc8df"
format_stamp: "Formatted at 2026-09-28 15:44:49 on dist-test-slave-j968"
server_key: "ad6a8973c9202acdcd71f7a482c14756"
server_key_iv: "67cdd04fab8e878eb9c57de4a2750d5b"
server_key_version: "encryptionkey@0"
I20260928 15:44:49.762557 19012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:44:49.766587 19027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:49.768384 19012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 15:44:49.768573 19012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "c12372d6ffdb4819ae77869af17dc8df"
format_stamp: "Formatted at 2026-09-28 15:44:49 on dist-test-slave-j968"
server_key: "ad6a8973c9202acdcd71f7a482c14756"
server_key_iv: "67cdd04fab8e878eb9c57de4a2750d5b"
server_key_version: "encryptionkey@0"
I20260928 15:44:49.768796 19012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:49.788149 19012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:49.792846 19012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:49.793148 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:49.794238 19012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:49.795641 19012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:49.795713 19012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:49.795883 19012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:49.795976 19012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 15:44:49.830631 19012 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:33467
I20260928 15:44:49.830724 19139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:33467 every 8 connection(s)
I20260928 15:44:49.832113 19012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:49.833832 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 19012
I20260928 15:44:49.834048 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:49.835100 13333 external_mini_cluster.cc:1428] Setting key 8740a359e30a00e7e75bdd8ea8eb6d7c
Sep 28 15:44:49 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610289, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
I20260928 15:44:49.860692 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:49.834958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:33697 (local address 127.13.5.126:43573)
0928 15:44:49.844367 (+ 9409us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:49.844375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:49.844403 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:44:49.844482 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:49.844488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:49.844575 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:49.844669 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:49.846760 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:49.848037 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:49.849750 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:49.850518 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:49.850672 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:49.854695 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:49.854701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:49.854705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:49.854741 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:49.857544 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:49.858267 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:49.858272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:49.858275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:49.858403 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:49.858811 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:49.858815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:49.858817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:49.859093 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:49.859206 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:49.860033 (+ 827us) server_negotiation.cc:299] Negotiation successful
0928 15:44:49.860291 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9228,"thread_start_us":9108,"threads_started":1}
I20260928 15:44:49.861881 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:49.835657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43573 (local address 127.13.5.65:33697)
0928 15:44:49.840732 (+ 5075us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:49.840819 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:49.844163 (+ 3344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:49.844996 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:49.845011 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:49.845877 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:49.846545 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:49.846562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:49.848214 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:49.848218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:49.849588 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:49.849600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:49.850751 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:49.850756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:49.850966 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:49.851698 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:49.851725 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:49.854429 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:49.857680 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:49.857687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:49.857700 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:49.858133 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:49.858522 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:49.858528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:49.858530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:49.858714 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:49.859204 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:49.859210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:49.859641 (+ 431us) client_negotiation.cc:769] Sending connection context
0928 15:44:49.859948 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 15:44:49.860606 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4635,"thread_start_us":4361,"threads_started":1}
I20260928 15:44:49.863481 19140 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43573
I20260928 15:44:49.863826 19140 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:49.864781 19140 heartbeater.cc:511] Master 127.13.5.126:43573 requested a full tablet report, sending...
I20260928 15:44:49.867514 18951 ts_manager.cc:194] Registered new tserver with Master: c12372d6ffdb4819ae77869af17dc8df (127.13.5.65:33467)
I20260928 15:44:49.869091 18951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:33697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:44:49.891305 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43573
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:50.226416 19146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:50.226780 19146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:50.226922 19146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:50.238528 19146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:50.238683 19146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:50.238852 19146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:50.249334 19146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:50.250639 19146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:50.252012 19146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:50.265609 19152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:50.265679 19154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:50.266000 19151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:50.267212 19146 server_base.cc:1034] running on GCE node
I20260928 15:44:50.267817 19146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:50.268828 19146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:50.270083 19146 hybrid_clock.cc:648] HybridClock initialized: now 1790610290270012 us; error 45 us; skew 500 ppm
I20260928 15:44:50.270419 19146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.273846 19146 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:44:50.275367 19146 webserver.cc:460] Webserver started at http://127.13.5.66:43353/ using document root <none> and password file <none>
I20260928 15:44:50.276135 19146 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:50.276274 19146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:50.276592 19146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:50.279783 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "e23ded14b8e24a4fb10009c293e133ca"
format_stamp: "Formatted at 2026-09-28 15:44:50 on dist-test-slave-j968"
server_key: "4c3c94b3629ead9d4b43adb0e4bfab1a"
server_key_iv: "93e2b9eb1f136e19ee0a59ecd1207d78"
server_key_version: "encryptionkey@0"
I20260928 15:44:50.280627 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "e23ded14b8e24a4fb10009c293e133ca"
format_stamp: "Formatted at 2026-09-28 15:44:50 on dist-test-slave-j968"
server_key: "4c3c94b3629ead9d4b43adb0e4bfab1a"
server_key_iv: "93e2b9eb1f136e19ee0a59ecd1207d78"
server_key_version: "encryptionkey@0"
I20260928 15:44:50.286051 19146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 15:44:50.289544 19161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:50.290939 19146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:44:50.291210 19146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "e23ded14b8e24a4fb10009c293e133ca"
format_stamp: "Formatted at 2026-09-28 15:44:50 on dist-test-slave-j968"
server_key: "4c3c94b3629ead9d4b43adb0e4bfab1a"
server_key_iv: "93e2b9eb1f136e19ee0a59ecd1207d78"
server_key_version: "encryptionkey@0"
I20260928 15:44:50.291427 19146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:50.321561 19146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:50.326558 19146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:50.326872 19146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:50.327961 19146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:50.329500 19146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:50.329607 19146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:50.329763 19146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:50.329843 19146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:50.363293 19146 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43799
I20260928 15:44:50.363339 19273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43799 every 8 connection(s)
I20260928 15:44:50.364699 19146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:50.372287 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 19146
I20260928 15:44:50.372480 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:50.373075 13333 external_mini_cluster.cc:1428] Setting key 6616be9948b487b76169879ace958130
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:50.386857 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:50.367299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:49951 (local address 127.13.5.126:43573)
0928 15:44:50.367767 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:50.367773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:50.368883 (+ 1110us) server_negotiation.cc:407] Connection header received
0928 15:44:50.370665 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:50.370669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:50.370740 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:50.370836 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:50.372654 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.373580 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.375552 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.376075 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.376209 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:50.379956 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:50.379962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:50.379965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:50.379998 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:50.383359 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:50.384222 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:50.384230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:50.384236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:50.384402 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:50.384827 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:50.384835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:50.384841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:50.385226 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:50.385379 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:50.386080 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 15:44:50.386344 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260928 15:44:50.387563 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:50.368032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43573 (local address 127.13.5.66:49951)
0928 15:44:50.368722 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:50.368765 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:50.370395 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:50.371057 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:50.371065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:50.371619 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:50.372469 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:50.372499 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.373735 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.373740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:50.375397 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:50.375412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.376302 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.376309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:50.376578 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:50.377275 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:50.377299 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:50.379689 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:50.383541 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:50.383549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:50.383565 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:50.384097 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:50.384517 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:50.384520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:50.384522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:50.384707 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:50.385352 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:50.385359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:50.385697 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 15:44:50.385925 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 15:44:50.386343 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":284,"threads_started":1}
I20260928 15:44:50.389149 19274 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43573
I20260928 15:44:50.389489 19274 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:50.390442 19274 heartbeater.cc:511] Master 127.13.5.126:43573 requested a full tablet report, sending...
I20260928 15:44:50.392210 18951 ts_manager.cc:194] Registered new tserver with Master: e23ded14b8e24a4fb10009c293e133ca (127.13.5.66:43799)
I20260928 15:44:50.393055 18951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:49951
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:44:50.422829 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43573
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:50.755946 19280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:50.756240 19280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:50.756453 19280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:50.763741 19280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:50.763909 19280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:50.764101 19280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:50.774600 19280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:50.776124 19280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:50.777477 19280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:50.791450 19288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:50.791503 19286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:50.791410 19285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:50.792352 19280 server_base.cc:1034] running on GCE node
I20260928 15:44:50.793083 19280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:50.794248 19280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:50.795506 19280 hybrid_clock.cc:648] HybridClock initialized: now 1790610290795444 us; error 41 us; skew 500 ppm
I20260928 15:44:50.795836 19280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.799345 19280 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:44:50.800942 19280 webserver.cc:460] Webserver started at http://127.13.5.67:46301/ using document root <none> and password file <none>
I20260928 15:44:50.801685 19280 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:50.801786 19280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:50.802052 19280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:50.805146 19280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "8da4917304f246ecbe07992d750cae63"
format_stamp: "Formatted at 2026-09-28 15:44:50 on dist-test-slave-j968"
server_key: "07cc3a873e7a1a5d634f2d44ef0a7341"
server_key_iv: "121845c08ef6bb026fec208ed3f336a0"
server_key_version: "encryptionkey@0"
I20260928 15:44:50.805930 19280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "8da4917304f246ecbe07992d750cae63"
format_stamp: "Formatted at 2026-09-28 15:44:50 on dist-test-slave-j968"
server_key: "07cc3a873e7a1a5d634f2d44ef0a7341"
server_key_iv: "121845c08ef6bb026fec208ed3f336a0"
server_key_version: "encryptionkey@0"
I20260928 15:44:50.811225 19280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:44:50.814937 19295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:50.816538 19280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:50.816674 19280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "8da4917304f246ecbe07992d750cae63"
format_stamp: "Formatted at 2026-09-28 15:44:50 on dist-test-slave-j968"
server_key: "07cc3a873e7a1a5d634f2d44ef0a7341"
server_key_iv: "121845c08ef6bb026fec208ed3f336a0"
server_key_version: "encryptionkey@0"
I20260928 15:44:50.816852 19280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:50.833004 19280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:50.837561 19280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:50.837874 19280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:50.838838 19280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:50.840284 19280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:50.840365 19280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:50.840509 19280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:50.840572 19280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:50.872933 19140 heartbeater.cc:503] Master 127.13.5.126:43573 was elected leader, sending a full tablet report...
I20260928 15:44:50.875080 19280 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:41509
I20260928 15:44:50.875205 19407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:41509 every 8 connection(s)
I20260928 15:44:50.876531 19280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:50.882794 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 19280
I20260928 15:44:50.882992 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:50.883549 13333 external_mini_cluster.cc:1428] Setting key 2de610ad145030774965076ec520596b
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:44:50.897400 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:50.879033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:43715 (local address 127.13.5.126:43573)
0928 15:44:50.879240 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:50.879246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:50.880704 (+ 1458us) server_negotiation.cc:407] Connection header received
0928 15:44:50.882563 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:50.882567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:50.882656 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:50.882748 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:50.884658 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.885605 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.887504 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.887958 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.888078 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:50.891359 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:50.891365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:50.891368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:50.891399 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:50.894108 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:50.894933 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:50.894940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:50.894944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:50.895084 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:50.895517 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:50.895522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:50.895538 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:50.895948 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:50.896104 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:50.896800 (+ 696us) server_negotiation.cc:299] Negotiation successful
0928 15:44:50.897066 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:44:50.898604 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:50.879698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43573 (local address 127.13.5.67:43715)
0928 15:44:50.880518 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:50.880582 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:50.882219 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:50.883035 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:50.883047 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:50.883757 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:50.884467 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:50.884485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.885788 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.885795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:50.887376 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:50.887388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.888155 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.888160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:50.888349 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:50.889005 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:50.889026 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:50.891142 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:50.894301 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:50.894311 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:50.894327 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:50.894815 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:50.895197 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:50.895202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:50.895206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:50.895400 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:50.896079 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:50.896086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:50.896430 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 15:44:50.896704 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 15:44:50.897257 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":339,"threads_started":1}
I20260928 15:44:50.899969 19408 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43573
I20260928 15:44:50.900287 19408 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:50.901011 19408 heartbeater.cc:511] Master 127.13.5.126:43573 requested a full tablet report, sending...
I20260928 15:44:50.902372 18951 ts_manager.cc:194] Registered new tserver with Master: 8da4917304f246ecbe07992d750cae63 (127.13.5.67:41509)
I20260928 15:44:50.903151 18951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:43715
I20260928 15:44:50.911439 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/test-admin.keytab.
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.938061 13333 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 15:44:50.943552 19415 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:44:50.940821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.13.5.65:33467)
0928 15:44:50.941498 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:50.941508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:50.941545 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:44:50.941634 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:50.941642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:50.941861 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:50.942041 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:50.943093 (+ 1052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50282: BlockingRecv error: recv got EOF from 127.0.0.1:50282 (error 108)
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":305,"threads_started":1}
W20260928 15:44:50.944012 13333 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.945573 13333 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.946941 13333 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:50.962136 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:50.948672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.13.5.65:33467)
0928 15:44:50.948856 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:50.948861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:50.948878 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:44:50.949040 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:50.949044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:50.949121 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:50.949202 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:50.950145 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.951190 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.952424 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.953044 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.953151 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:50.956690 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:50.956701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:50.956717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:50.956749 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:50.959626 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:50.960324 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:50.960331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:50.960334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:50.960452 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:50.960768 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:50.960771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:50.960773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:50.961118 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:50.961363 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:50.961683 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 15:44:50.961885 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 15:44:50.968317 19415 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:44:50.966228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.13.5.65:33467)
0928 15:44:50.966442 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:50.966451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:50.966804 (+ 353us) server_negotiation.cc:407] Connection header received
0928 15:44:50.966981 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:50.966986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:50.967068 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:50.967170 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:50.968054 (+ 884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50292: BlockingRecv error: recv got EOF from 127.0.0.1:50292 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 15:44:50.968351 13333 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.969852 13333 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.971118 13333 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:50.985847 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:50.972532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.13.5.65:33467)
0928 15:44:50.972705 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:50.972709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:50.972983 (+ 274us) server_negotiation.cc:407] Connection header received
0928 15:44:50.973173 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:50.973178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:50.973243 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:50.973321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:50.974297 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.975114 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.976348 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.976794 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.976889 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:50.980385 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:50.980392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:50.980397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:50.980441 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:50.983273 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:50.984001 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:50.984005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:50.984007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:50.984130 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:50.984471 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:50.984475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:50.984476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:50.984761 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:50.985033 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:50.985362 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 15:44:50.985578 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 15:44:50.990362 19415 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:44:50.988570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.13.5.65:33467)
0928 15:44:50.988777 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:50.988783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:50.988990 (+ 207us) server_negotiation.cc:407] Connection header received
0928 15:44:50.989188 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:50.989192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:50.989256 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:50.989354 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:50.990127 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50296: BlockingRecv error: recv got EOF from 127.0.0.1:50296 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 15:44:50.990551 13333 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610238672534-13333-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.992123 13333 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:50.993405 13333 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:44:50 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:44:51.007747 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:50.994719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.13.5.65:33467)
0928 15:44:50.994920 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:50.994927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:50.995352 (+ 425us) server_negotiation.cc:407] Connection header received
0928 15:44:50.995573 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:50.995579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:50.995672 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:50.995790 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:50.996846 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.997552 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.998726 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:50.999122 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:50.999214 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:51.002844 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:51.002849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:51.002852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:51.002886 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:51.005512 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:51.006112 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:51.006116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:51.006118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:51.006259 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:51.006571 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:51.006574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:51.006576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:51.006855 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:51.007081 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:51.007284 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 15:44:51.007483 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:44:51.009464 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 19012
I20260928 15:44:51.020748 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:51.010129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.13.5.65:33467)
0928 15:44:51.010280 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:51.010283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:51.010625 (+ 342us) server_negotiation.cc:407] Connection header received
0928 15:44:51.010734 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:51.010737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:51.010787 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:51.010860 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:51.011735 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:51.012468 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:51.013646 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:51.014115 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:51.014221 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:51.016163 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:51.016167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:51.016170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:51.016199 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:51.018623 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:51.019138 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:51.019141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:51.019143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:51.019267 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:51.019569 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:51.019572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:51.019573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:51.019813 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:51.020093 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:51.020286 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 15:44:51.020457 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:44:51.021368 19135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:51.107697 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 19012
I20260928 15:44:51.124743 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 19146
Sep 28 15:44:51 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:44:51.140229 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:51.125725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.13.5.66:43799)
0928 15:44:51.126281 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:51.126288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:51.126314 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:44:51.126377 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:51.126384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:51.126639 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:51.126775 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:51.127717 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:51.128837 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:51.130123 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:51.130748 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:51.130896 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:51.134761 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:51.134771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:51.134786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:51.134821 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:51.137620 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:51.138305 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:51.138311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:51.138313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:51.138428 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:51.138789 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:51.138792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:51.138794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:51.139116 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:51.139336 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:51.139651 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 15:44:51.139847 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":276,"threads_started":1}
I20260928 15:44:51.141613 19269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:51.225744 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 19146
I20260928 15:44:51.242452 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 19280
Sep 28 15:44:51 dist-test-slave-j968 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:44:51.259073 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:51.243728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.13.5.67:41509)
0928 15:44:51.244560 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:51.244567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:51.244596 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:44:51.244662 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:51.244668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:51.244889 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:51.245029 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:51.246052 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:51.247083 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:51.248330 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:51.248877 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:51.248974 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:51.253296 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:51.253307 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:51.253323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:51.253356 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:51.256422 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:51.257202 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:51.257209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:51.257211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:51.257325 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:51.257731 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:51.257735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:51.257737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:51.258077 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:51.258310 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:51.258578 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 15:44:51.258780 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":455,"threads_started":1}
I20260928 15:44:51.260552 19403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:51.344668 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 19280
I20260928 15:44:51.361232 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 18919
I20260928 15:44:51.362636 18981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:51.467787 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 18919
2026-09-28T15:44:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4009 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:44:51 dist-test-slave-j968 krb5kdc[19435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:44:51 dist-test-slave-j968 krb5kdc[19435](info): set up 2 sockets
Sep 28 15:44:51 dist-test-slave-j968 krb5kdc[19435](info): commencing operation
krb5kdc: starting...
W20260928 15:44:52.603230 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:44:52 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:44:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.126.keytab.
Entry for principal oryx/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.126.keytab.
I20260928 15:44:52.821686 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36553
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.126.keytab
--principal=oryx/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:36553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:53.154680 19448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:53.154987 19448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:53.155078 19448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:53.162436 19448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:53.162557 19448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:53.162604 19448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:53.162654 19448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:53.162699 19448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:53.172868 19448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:36553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.126.keytab
--principal=oryx/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:53.174235 19448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:53.175563 19448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:53.188004 19453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:53.188088 19454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:53.188047 19456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:53.189215 19448 server_base.cc:1034] running on GCE node
I20260928 15:44:53.190198 19448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:53.191761 19448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:53.193001 19448 hybrid_clock.cc:648] HybridClock initialized: now 1790610293192950 us; error 33 us; skew 500 ppm
I20260928 15:44:53.193387 19448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:53 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610293, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:53.202169 19448 init.cc:377] Logged in from keytab as oryx/127.13.5.126@KRBTEST.COM (short username oryx)
I20260928 15:44:53.203948 19448 webserver.cc:460] Webserver started at http://127.13.5.126:39445/ using document root <none> and password file <none>
I20260928 15:44:53.204802 19448 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:53.204967 19448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:53.205303 19448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:53.208649 19448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "e2722471774848d0923a854406fc2ba2"
format_stamp: "Formatted at 2026-09-28 15:44:53 on dist-test-slave-j968"
server_key: "58b5f89996d71505204ecb6f10253219"
server_key_iv: "7fe76a60a82d290f78e1f03dfa121dae"
server_key_version: "encryptionkey@0"
I20260928 15:44:53.209492 19448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "e2722471774848d0923a854406fc2ba2"
format_stamp: "Formatted at 2026-09-28 15:44:53 on dist-test-slave-j968"
server_key: "58b5f89996d71505204ecb6f10253219"
server_key_iv: "7fe76a60a82d290f78e1f03dfa121dae"
server_key_version: "encryptionkey@0"
I20260928 15:44:53.215095 19448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:44:53.219033 19463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:53.220721 19448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:53.220933 19448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "e2722471774848d0923a854406fc2ba2"
format_stamp: "Formatted at 2026-09-28 15:44:53 on dist-test-slave-j968"
server_key: "58b5f89996d71505204ecb6f10253219"
server_key_iv: "7fe76a60a82d290f78e1f03dfa121dae"
server_key_version: "encryptionkey@0"
I20260928 15:44:53.221158 19448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:53.242223 19448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:53.246521 19448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:53.246834 19448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:53.266120 19448 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:36553
I20260928 15:44:53.266157 19514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:36553 every 8 connection(s)
I20260928 15:44:53.267594 19448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:44:53.271431 19515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:53.272987 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 19448
I20260928 15:44:53.273216 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:44:53.274082 13333 external_mini_cluster.cc:1428] Setting key 729fd2b3bcfd3f2f0a64e1453a0f1833
I20260928 15:44:53.287045 19515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: Bootstrap starting.
Sep 28 15:44:53 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:44:53.291311 19515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:53.292578 19515 log.cc:824] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:53.296218 19515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: No bootstrap required, opened a new log
I20260928 15:44:53.301191 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:53.276309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.13.5.126:36553)
0928 15:44:53.277137 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:53.277149 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:53.277196 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:44:53.278825 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:53.278850 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:53.279287 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:53.279670 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:53.281633 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:53.283156 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:53.285072 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:53.286014 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:53.286168 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:53.291235 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:53.291245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:53.291262 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:53.291314 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:53.295183 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:53.296165 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:53.296177 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:53.296188 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:53.296410 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:53.297425 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:53.297431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:53.297434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:53.298295 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:53.298633 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:53.299116 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 15:44:53.299631 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":375,"threads_started":1}
I20260928 15:44:53.306947 19515 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } }
I20260928 15:44:53.307307 19515 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:53.307371 19515 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2722471774848d0923a854406fc2ba2, State: Initialized, Role: FOLLOWER
I20260928 15:44:53.308043 19515 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } }
I20260928 15:44:53.308248 19515 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:44:53.308418 19515 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:44:53.308631 19515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:53.310927 19515 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } }
I20260928 15:44:53.311403 19515 leader_election.cc:302] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2722471774848d0923a854406fc2ba2; no voters:
I20260928 15:44:53.312093 19515 leader_election.cc:288] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:44:53.312268 19520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:53.312980 19520 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 1 LEADER]: Becoming Leader. State: Replica: e2722471774848d0923a854406fc2ba2, State: Running, Role: LEADER
I20260928 15:44:53.313947 19520 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } }
I20260928 15:44:53.314464 19515 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:53.316970 19522 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2722471774848d0923a854406fc2ba2. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } }
I20260928 15:44:53.317346 19522 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:53.317873 19521 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } }
I20260928 15:44:53.318245 19521 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:53.318461 19529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:44:53.326025 19529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:44:53.338212 19529 catalog_manager.cc:1347] Generated new cluster ID: 61c2b26ae621471b90884dc025292186
I20260928 15:44:53.338346 19529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:44:53.363960 19529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:44:53.366104 19529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:44:53.376269 19529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: Generated new TSK 0
I20260928 15:44:53.377390 19529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab.
Entry for principal oryx/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab.
I20260928 15:44:53.432716 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36553
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab
--principal=oryx/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:53.772248 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:53.772543 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:53.772708 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:53.780040 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:53.780158 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:53.780344 19541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:53.790544 19541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab
--principal=oryx/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:53.792097 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:53.793493 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:53.807761 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:53.807726 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:53.807804 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:53.808737 19541 server_base.cc:1034] running on GCE node
I20260928 15:44:53.809369 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:53.810326 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:53.811555 19541 hybrid_clock.cc:648] HybridClock initialized: now 1790610293811503 us; error 35 us; skew 500 ppm
I20260928 15:44:53.811995 19541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:53 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610293, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:53.815328 19541 init.cc:377] Logged in from keytab as oryx/127.13.5.65@KRBTEST.COM (short username oryx)
I20260928 15:44:53.817057 19541 webserver.cc:460] Webserver started at http://127.13.5.65:34209/ using document root <none> and password file <none>
I20260928 15:44:53.817773 19541 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:53.817902 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:53.818221 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:53.821530 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "5b75b8b58bd4425499d56d72b7158364"
format_stamp: "Formatted at 2026-09-28 15:44:53 on dist-test-slave-j968"
server_key: "d60df972dd57241a4747e61fdadb155f"
server_key_iv: "950996194171f5e6e81701c841668511"
server_key_version: "encryptionkey@0"
I20260928 15:44:53.822358 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "5b75b8b58bd4425499d56d72b7158364"
format_stamp: "Formatted at 2026-09-28 15:44:53 on dist-test-slave-j968"
server_key: "d60df972dd57241a4747e61fdadb155f"
server_key_iv: "950996194171f5e6e81701c841668511"
server_key_version: "encryptionkey@0"
I20260928 15:44:53.828105 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:44:53.831645 19556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:53.833257 19541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:53.833426 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "5b75b8b58bd4425499d56d72b7158364"
format_stamp: "Formatted at 2026-09-28 15:44:53 on dist-test-slave-j968"
server_key: "d60df972dd57241a4747e61fdadb155f"
server_key_iv: "950996194171f5e6e81701c841668511"
server_key_version: "encryptionkey@0"
I20260928 15:44:53.833613 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:53.852211 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:53.857012 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:53.857348 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:53.858388 19541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:53.859910 19541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:53.859985 19541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:53.860126 19541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:53.860186 19541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:53.892791 19541 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:33255
I20260928 15:44:53.892840 19668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:33255 every 8 connection(s)
I20260928 15:44:53.894155 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:53.904503 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 19541
I20260928 15:44:53.904767 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:44:53.905637 13333 external_mini_cluster.cc:1428] Setting key fc27d358f77d0e306d6dcc35f0f13f75
Sep 28 15:44:53 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610293, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:44:53.918946 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:53.896857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:58225 (local address 127.13.5.126:36553)
0928 15:44:53.897334 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:53.897340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:53.898332 (+ 992us) server_negotiation.cc:407] Connection header received
0928 15:44:53.900203 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:53.900207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:53.900276 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:53.900464 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:53.902191 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:53.903017 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:53.905066 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:53.905781 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:53.905946 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:53.910756 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:53.910766 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:53.910776 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:53.910828 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:53.914788 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:53.915844 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:53.915852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:53.915859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:53.916060 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:53.916527 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:53.916534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:53.916539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:53.916981 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:53.917161 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:53.918217 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0928 15:44:53.918490 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":217,"threads_started":1}
I20260928 15:44:53.920658 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:53.897459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.65:58225)
0928 15:44:53.898172 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:53.898210 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:53.899936 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:53.900679 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:53.900688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:53.901292 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:53.902001 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:53.902022 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:53.903181 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:53.903185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:53.904859 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:53.904878 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:53.906053 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:53.906059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:53.906303 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:53.907267 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:53.907308 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:53.910484 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:53.914960 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:53.914968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:53.914986 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:53.915699 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:53.916183 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:53.916187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:53.916189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:53.916386 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:53.917141 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:53.917148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:53.917706 (+ 558us) client_negotiation.cc:769] Sending connection context
0928 15:44:53.918082 (+ 376us) client_negotiation.cc:240] Negotiation successful
0928 15:44:53.918794 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":268,"threads_started":1}
I20260928 15:44:53.922632 19669 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36553
I20260928 15:44:53.922971 19669 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:53.923775 19669 heartbeater.cc:511] Master 127.13.5.126:36553 requested a full tablet report, sending...
I20260928 15:44:53.926424 19480 ts_manager.cc:194] Registered new tserver with Master: 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255)
I20260928 15:44:53.929065 19480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:58225
WARNING: no policy specified for oryx/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab.
Entry for principal oryx/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab.
I20260928 15:44:53.960002 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36553
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab
--principal=oryx/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:54.297569 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:54.297855 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:54.297955 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:54.305600 19675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:54.305721 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:54.305835 19675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:54.316524 19675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab
--principal=oryx/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:54.317857 19675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:54.319308 19675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:54.333482 19680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:54.333503 19683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:54.333446 19681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:54.334336 19675 server_base.cc:1034] running on GCE node
I20260928 15:44:54.334964 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:54.335850 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:54.337149 19675 hybrid_clock.cc:648] HybridClock initialized: now 1790610294337080 us; error 47 us; skew 500 ppm
I20260928 15:44:54.337522 19675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:54 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610294, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:54.340931 19675 init.cc:377] Logged in from keytab as oryx/127.13.5.66@KRBTEST.COM (short username oryx)
I20260928 15:44:54.342551 19675 webserver.cc:460] Webserver started at http://127.13.5.66:42023/ using document root <none> and password file <none>
I20260928 15:44:54.343269 19675 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:54.343415 19675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:54.343722 19675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:54.346725 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "dfbdcc4f879041f8953c4b203a208781"
format_stamp: "Formatted at 2026-09-28 15:44:54 on dist-test-slave-j968"
server_key: "1bb67e0c2dc00c3a73570bbde8db26d7"
server_key_iv: "a431f9ee3564c6e1e13593d1a4476cbc"
server_key_version: "encryptionkey@0"
I20260928 15:44:54.347512 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "dfbdcc4f879041f8953c4b203a208781"
format_stamp: "Formatted at 2026-09-28 15:44:54 on dist-test-slave-j968"
server_key: "1bb67e0c2dc00c3a73570bbde8db26d7"
server_key_iv: "a431f9ee3564c6e1e13593d1a4476cbc"
server_key_version: "encryptionkey@0"
I20260928 15:44:54.353418 19675 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 15:44:54.357111 19690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:54.358548 19675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:54.358762 19675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "dfbdcc4f879041f8953c4b203a208781"
format_stamp: "Formatted at 2026-09-28 15:44:54 on dist-test-slave-j968"
server_key: "1bb67e0c2dc00c3a73570bbde8db26d7"
server_key_iv: "a431f9ee3564c6e1e13593d1a4476cbc"
server_key_version: "encryptionkey@0"
I20260928 15:44:54.358978 19675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:54.391281 19675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:54.396034 19675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:54.396347 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:54.397382 19675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:54.398790 19675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:54.398864 19675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:54.399042 19675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:54.399093 19675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:54.432533 19675 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43093
I20260928 15:44:54.432621 19802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43093 every 8 connection(s)
I20260928 15:44:54.433905 19675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:54.441609 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 19675
I20260928 15:44:54.441812 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:44:54.442370 13333 external_mini_cluster.cc:1428] Setting key 319c542607ea2610597d2197c2f10cfd
Sep 28 15:44:54 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610294, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:44:54.456600 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:54.436707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:50171 (local address 127.13.5.126:36553)
0928 15:44:54.437228 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:54.437232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:54.438359 (+ 1127us) server_negotiation.cc:407] Connection header received
0928 15:44:54.440303 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:54.440308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:54.440401 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:54.440529 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:54.442611 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:54.443473 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:54.445757 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:54.446202 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:54.446307 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:54.450600 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:54.450605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:54.450609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:54.450643 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:54.453409 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:54.454163 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:54.454167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:54.454170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:54.454293 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:54.454793 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:54.454798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:54.454801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:54.455069 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:54.455226 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:54.455967 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 15:44:54.456193 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":312,"threads_started":1}
I20260928 15:44:54.457634 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:54.437291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.66:50171)
0928 15:44:54.438158 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:54.438207 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:54.440029 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:54.440770 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:54.440779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:54.441581 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:54.442419 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:54.442446 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:54.443639 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:54.443645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:54.445582 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:54.445599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:54.446444 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:54.446454 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:54.446781 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:54.447553 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:54.447593 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:54.450315 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:54.453560 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:54.453569 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:54.453588 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:54.454019 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:54.454476 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:54.454488 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:54.454491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:54.454670 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:54.455198 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:54.455206 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:54.455569 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 15:44:54.455840 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 15:44:54.456442 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":393,"threads_started":1}
I20260928 15:44:54.459220 19803 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36553
I20260928 15:44:54.459630 19803 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:54.460490 19803 heartbeater.cc:511] Master 127.13.5.126:36553 requested a full tablet report, sending...
I20260928 15:44:54.462110 19480 ts_manager.cc:194] Registered new tserver with Master: dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093)
WARNING: no policy specified for oryx/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:44:54.463402 19480 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:50171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab.
Entry for principal oryx/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab.
I20260928 15:44:54.492033 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36553
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab
--principal=oryx/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:54.830327 19809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:54.830644 19809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:54.830752 19809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:54.839049 19809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:54.839176 19809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:54.839301 19809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:54.849807 19809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab
--principal=oryx/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:54.851106 19809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:54.852548 19809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:54.866173 19817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:54.866272 19815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:54.866276 19814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:54.867002 19809 server_base.cc:1034] running on GCE node
I20260928 15:44:54.867626 19809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:54.868659 19809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:54.869865 19809 hybrid_clock.cc:648] HybridClock initialized: now 1790610294869782 us; error 75 us; skew 500 ppm
I20260928 15:44:54.870190 19809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:54 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610294, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:54.873490 19809 init.cc:377] Logged in from keytab as oryx/127.13.5.67@KRBTEST.COM (short username oryx)
I20260928 15:44:54.874959 19809 webserver.cc:460] Webserver started at http://127.13.5.67:44669/ using document root <none> and password file <none>
I20260928 15:44:54.875663 19809 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:54.875793 19809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:54.876132 19809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:54.879011 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "c830cd92804c4c96bf17a480cf896d6d"
format_stamp: "Formatted at 2026-09-28 15:44:54 on dist-test-slave-j968"
server_key: "b3de6fae21c984f1cbd99da75dcdd654"
server_key_iv: "1b3581c6be8d88cca85fd665e87b34f0"
server_key_version: "encryptionkey@0"
I20260928 15:44:54.879801 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "c830cd92804c4c96bf17a480cf896d6d"
format_stamp: "Formatted at 2026-09-28 15:44:54 on dist-test-slave-j968"
server_key: "b3de6fae21c984f1cbd99da75dcdd654"
server_key_iv: "1b3581c6be8d88cca85fd665e87b34f0"
server_key_version: "encryptionkey@0"
I20260928 15:44:54.885097 19809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:44:54.888743 19824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:54.890241 19809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:54.890429 19809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "c830cd92804c4c96bf17a480cf896d6d"
format_stamp: "Formatted at 2026-09-28 15:44:54 on dist-test-slave-j968"
server_key: "b3de6fae21c984f1cbd99da75dcdd654"
server_key_iv: "1b3581c6be8d88cca85fd665e87b34f0"
server_key_version: "encryptionkey@0"
I20260928 15:44:54.890642 19809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:54.913045 19809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:54.917518 19809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:54.917824 19809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:54.918936 19809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:54.920485 19809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:54.920590 19809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:54.920748 19809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:54.920820 19809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:54.932927 19669 heartbeater.cc:503] Master 127.13.5.126:36553 was elected leader, sending a full tablet report...
I20260928 15:44:54.955519 19809 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:43295
I20260928 15:44:54.955571 19936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:43295 every 8 connection(s)
I20260928 15:44:54.956889 19809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:54.964131 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 19809
I20260928 15:44:54.964315 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:44:54.964779 13333 external_mini_cluster.cc:1428] Setting key 99f445840be3aedbe1f3b78d77e7fc7e
Sep 28 15:44:54 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610294, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:44:54.978283 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:54.959730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:37253 (local address 127.13.5.126:36553)
0928 15:44:54.960215 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:54.960220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:54.961381 (+ 1161us) server_negotiation.cc:407] Connection header received
0928 15:44:54.963407 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:54.963412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:54.963518 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:54.963666 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:54.965508 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:54.966533 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:54.968697 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:54.969111 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:54.969213 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:54.972671 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:54.972677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:54.972681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:54.972729 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:54.975270 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:54.976160 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:54.976164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:54.976167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:54.976282 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:54.976740 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:54.976743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:54.976745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:54.977011 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:54.977129 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:54.977829 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 15:44:54.978007 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
I20260928 15:44:54.979453 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:54.960452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.67:37253)
0928 15:44:54.961194 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:54.961240 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:54.963121 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:54.963988 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:54.963997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:54.964624 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:54.965298 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:54.965330 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:54.966740 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:54.966746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:54.968553 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:54.968566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:54.969302 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:54.969306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:54.969505 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:54.970204 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:54.970228 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:54.972438 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:54.975486 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:54.975495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:54.975509 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:54.976046 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:54.976374 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:54.976377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:54.976379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:54.976628 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:54.977115 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:54.977121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:54.977491 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 15:44:54.977727 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 15:44:54.978116 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":274,"threads_started":1}
I20260928 15:44:54.981124 19937 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36553
I20260928 15:44:54.981449 19937 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:54.982175 19937 heartbeater.cc:511] Master 127.13.5.126:36553 requested a full tablet report, sending...
I20260928 15:44:54.983698 19479 ts_manager.cc:194] Registered new tserver with Master: c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295)
I20260928 15:44:54.984664 19479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:37253
I20260928 15:44:54.993363 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.5.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.125.keytab.
Entry for principal oryx/127.13.5.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.125.keytab.
I20260928 15:44:55.045182 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.125:45387
--webserver_interface=127.13.5.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.125.keytab
--principal=oryx/127.13.5.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:36553,127.13.5.125:45387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:44:55.408981 19943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:55.409268 19943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:55.409458 19943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:55.416934 19943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:55.417042 19943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:55.417141 19943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:55.417236 19943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:55.417315 19943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:55.427837 19943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:36553,127.13.5.125:45387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.125.keytab
--principal=oryx/127.13.5.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.125:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.5.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
W20260928 15:44:55.428294 19943 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.5.126:36553,127.13.5.125:45387'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 15:44:55.429412 19943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:55.430791 19943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:55.443261 19949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:55.443317 19951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:55.443333 19948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:55.444074 19943 server_base.cc:1034] running on GCE node
I20260928 15:44:55.444800 19943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:55.446139 19943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:55.447439 19943 hybrid_clock.cc:648] HybridClock initialized: now 1790610295447349 us; error 47 us; skew 500 ppm
I20260928 15:44:55.447844 19943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:55 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610295, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:55.451344 19943 init.cc:377] Logged in from keytab as oryx/127.13.5.125@KRBTEST.COM (short username oryx)
I20260928 15:44:55.452956 19943 webserver.cc:460] Webserver started at http://127.13.5.125:36145/ using document root <none> and password file <none>
I20260928 15:44:55.453752 19943 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:55.453898 19943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:55.454262 19943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:55.457543 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data/instance:
uuid: "ba69234c12ab4e909a9e6bfc2b2843fe"
format_stamp: "Formatted at 2026-09-28 15:44:55 on dist-test-slave-j968"
server_key: "c8e9a451a6e588790d5764d14aef9be2"
server_key_iv: "1cd00b17269988293581b1590577d810"
server_key_version: "encryptionkey@0"
I20260928 15:44:55.458391 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal/instance:
uuid: "ba69234c12ab4e909a9e6bfc2b2843fe"
format_stamp: "Formatted at 2026-09-28 15:44:55 on dist-test-slave-j968"
server_key: "c8e9a451a6e588790d5764d14aef9be2"
server_key_iv: "1cd00b17269988293581b1590577d810"
server_key_version: "encryptionkey@0"
I20260928 15:44:55.464443 19943 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 15:44:55.466446 19803 heartbeater.cc:503] Master 127.13.5.126:36553 was elected leader, sending a full tablet report...
I20260928 15:44:55.468292 19958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:55.469781 19943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:55.469986 19943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
uuid: "ba69234c12ab4e909a9e6bfc2b2843fe"
format_stamp: "Formatted at 2026-09-28 15:44:55 on dist-test-slave-j968"
server_key: "c8e9a451a6e588790d5764d14aef9be2"
server_key_iv: "1cd00b17269988293581b1590577d810"
server_key_version: "encryptionkey@0"
I20260928 15:44:55.470213 19943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:55.485958 19943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:55.490162 19943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:55.490440 19943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:55.509909 19943 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.125:45387
I20260928 15:44:55.509931 20009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.125:45387 every 8 connection(s)
I20260928 15:44:55.511335 19943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
I20260928 15:44:55.515424 20010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:55.516685 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 19943
I20260928 15:44:55.525373 20010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:44:55 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.125@KRBTEST.COM
Sep 28 15:44:55 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610295, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.125@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:44:55.543702 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.519314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.13.5.125:45387)
0928 15:44:55.520161 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:55.520176 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:55.520218 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:44:55.522129 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:55.522152 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:55.522563 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:55.522914 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:55.524812 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.526463 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.528275 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.529440 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.529740 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.534433 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:55.534443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:55.534458 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:55.534513 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:55.538156 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.539111 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.539120 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.539131 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.539347 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.539795 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.539803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.539809 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.540752 (+ 943us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:55.541103 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:55.541698 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 15:44:55.542193 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":347,"threads_started":1}
I20260928 15:44:55.546116 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.528303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.13.5.126:36553)
0928 15:44:55.528856 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:55.528862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:55.529521 (+ 659us) server_negotiation.cc:407] Connection header received
0928 15:44:55.529771 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:55.529775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:55.529846 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:55.530033 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:55.531203 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.532097 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.534190 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.534902 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.535061 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.539151 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:55.539158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:55.539163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:55.539211 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:55.543139 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.543945 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.543950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.543954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.544116 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.544568 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.544574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.544577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.544916 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:55.545079 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:55.545532 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 15:44:55.545764 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":310,"threads_started":1}
I20260928 15:44:55.546211 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.528305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:55810)
0928 15:44:55.529358 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:55.529404 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:55.529581 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:55.530091 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:55.530099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:55.530764 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:55.531031 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.531040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.532230 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.532238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.534035 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.534048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.535126 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.535131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.535363 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.536083 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:55.536107 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:55.538926 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:55.543284 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.543293 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.543298 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.543748 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.544243 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.544247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.544250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.544421 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.545064 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:55.545071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:55.545236 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 15:44:55.545451 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 15:44:55.545758 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":638,"threads_started":1}
I20260928 15:44:55.548295 20010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:44:55 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610295, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.125@KRBTEST.COM for oryx/127.13.5.125@KRBTEST.COM
I20260928 15:44:55.560642 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.548981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.0.0.1:58816)
0928 15:44:55.549163 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:55.549179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:55.549291 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:55.549613 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:55.549617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:55.549972 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:55.550096 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.550103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.551064 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.551069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.551967 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.551978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.552653 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.552658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.552935 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.553502 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:55.553519 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:55.555410 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:55.558295 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.558301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.558305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.558653 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.558989 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.558996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.559001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.559123 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.559630 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:55.559635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:55.559785 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 15:44:55.559968 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 15:44:55.560260 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":56}
I20260928 15:44:55.560690 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.549175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.13.5.125:45387)
0928 15:44:55.549312 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:55.549317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:55.549337 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:44:55.549445 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:55.549448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:55.549499 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:55.549599 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:55.550223 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.550924 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.552101 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.552506 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.552643 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.555571 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:55.555583 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:55.555586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:55.555612 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:55.558150 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.558763 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.558770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.558773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.558884 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.559220 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.559227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.559231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.559517 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:55.559677 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:55.560057 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 15:44:55.560397 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":48}
I20260928 15:44:55.566114 20010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Bootstrap starting.
I20260928 15:44:55.569245 20010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:55.570240 20010 log.cc:824] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:55.573024 20010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: No bootstrap required, opened a new log
I20260928 15:44:55.580259 20010 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } }
I20260928 15:44:55.580610 20010 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:55.580681 20010 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba69234c12ab4e909a9e6bfc2b2843fe, State: Initialized, Role: FOLLOWER
I20260928 15:44:55.581192 20010 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } }
I20260928 15:44:55.582067 20017 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } } }
I20260928 15:44:55.582441 20017 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:44:55.583154 20010 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:44:55.592628 20028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:44:55.592828 20028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:44:55.594822 19943 master_runner.cc:428] Detected that this master ba69234c12ab4e909a9e6bfc2b2843fe is joining an existing cluster
I20260928 15:44:55.594935 19943 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.5.125:45387
I20260928 15:44:55.626468 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.600059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.13.5.126:36553)
0928 15:44:55.600232 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:55.600239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:55.609491 (+ 9252us) server_negotiation.cc:407] Connection header received
0928 15:44:55.609715 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:55.609723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:55.609861 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:55.609964 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:55.611526 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.612436 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.614787 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.615494 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.615600 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.618263 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:55.618271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:55.618277 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:55.618319 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:55.623027 (+ 4708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.623956 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.623963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.623968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.624137 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.624649 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.624655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.624658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.625048 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:55.625208 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:55.625810 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 15:44:55.626094 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:44:55.627573 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.599597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.13.5.125:45387)
0928 15:44:55.599821 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:55.599829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:55.612249 (+ 12420us) server_negotiation.cc:407] Connection header received
0928 15:44:55.612402 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:55.612407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:55.612493 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:55.612588 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:55.613952 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.615059 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.617059 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.617849 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.617950 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.620611 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:55.620623 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:55.620630 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:55.620669 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:55.624403 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.625212 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.625218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.625223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.625364 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.625907 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.625912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.625914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.626258 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:55.626403 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:55.626952 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 15:44:55.627188 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":66}
I20260928 15:44:55.628000 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.599557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:55826)
0928 15:44:55.609316 (+ 9759us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:55.609343 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:55.609557 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:55.610178 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:55.610185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:55.610930 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:55.611348 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.611359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.612601 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.612606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.614622 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.614636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.615927 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.615934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.616121 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.616848 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:55.616872 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:55.618064 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:55.623235 (+ 5171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.623240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.623243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.623765 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.624334 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.624339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.624343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.624515 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.625283 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:55.625290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:55.625509 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 15:44:55.627161 (+ 1652us) client_negotiation.cc:240] Negotiation successful
0928 15:44:55.627454 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9591,"mutex_wait_us":3,"thread_start_us":12808,"threads_started":1}
WARNING: no policy specified for oryx/127.13.5.124@KRBTEST.COM; defaulting to no policy
I20260928 15:44:55.629706 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.600123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.0.0.1:58832)
0928 15:44:55.612133 (+ 12010us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:55.612152 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:55.612284 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:55.612663 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:55.612667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:55.613494 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:55.613811 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.613822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.615283 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.615289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.616898 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.616914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.618229 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.618236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.618411 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.619108 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:55.619173 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:55.620410 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:55.624545 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.624552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.624559 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.625098 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.625628 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.625635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.625640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.625802 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.626484 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:55.626491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:55.626715 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 15:44:55.628084 (+ 1369us) client_negotiation.cc:240] Negotiation successful
0928 15:44:55.628409 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11872,"thread_start_us":8867,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.124@KRBTEST.COM" created.
Sep 28 15:44:55 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610293, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.126@KRBTEST.COM for oryx/127.13.5.125@KRBTEST.COM
I20260928 15:44:55.655959 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.635812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58844 (local address 127.13.5.125:45387)
0928 15:44:55.636049 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:55.636057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:55.641865 (+ 5808us) server_negotiation.cc:407] Connection header received
0928 15:44:55.642303 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:55.642309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:55.642428 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:55.642528 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:55.643908 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.644630 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.645995 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.646473 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.646621 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.650085 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:55.650090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:55.650093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:55.650126 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:55.653039 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.653777 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.653783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.653788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.653929 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.654330 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.654335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.654339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.654655 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:55.654824 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:55.655326 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 15:44:55.655548 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 15:44:55.656014 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.635772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.0.0.1:58844)
0928 15:44:55.641675 (+ 5903us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:55.641733 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:55.641996 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:55.642755 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:55.642763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:55.643475 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:55.643710 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.643721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.644773 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.644778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.645826 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.645850 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.646650 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.646655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.646861 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.647574 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:55.647591 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:55.649888 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:55.653199 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.653212 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.653220 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.653650 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.654046 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.654051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.654055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.654219 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.654797 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:55.654806 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:55.655003 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 15:44:55.655210 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 15:44:55.655548 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5682,"thread_start_us":6468,"threads_started":1}
I20260928 15:44:55.658408 19479 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.5.125:45387: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe"
member_type: NON_VOTER
last_known_addr {
host: "127.13.5.125"
port: 45387
}
attrs {
promote: true
}
}
dest_uuid: "e2722471774848d0923a854406fc2ba2"
cas_config_opid_index: -1
I20260928 15:44:55.659643 19479 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } }
I20260928 15:44:55.663129 20041 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ba69234c12ab4e909a9e6bfc2b2843fe (127.13.5.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } }
I20260928 15:44:55.664386 19985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.124.keytab.
Entry for principal oryx/127.13.5.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.124.keytab.
I20260928 15:44:55.667078 20041 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.5.125:45387
I20260928 15:44:55.667845 19943 master.cc:537] Master@127.13.5.125:45387 shutting down...
I20260928 15:44:55.668010 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.124:33123
--webserver_interface=127.13.5.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.124.keytab
--principal=oryx/127.13.5.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:36553,127.13.5.125:45387,127.13.5.124:33123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:44:55.668125 20040 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } }
I20260928 15:44:55.668424 20040 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:55.677068 19985 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Refusing update from remote peer e2722471774848d0923a854406fc2ba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 15:44:55.678416 20041 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:44:55.682569 19467 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 15:44:55.684340 19467 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 -> Peer ba69234c12ab4e909a9e6bfc2b2843fe (127.13.5.125:45387): Couldn't send request to peer ba69234c12ab4e909a9e6bfc2b2843fe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 15:44:55.718631 19943 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:55.719141 19943 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:55.719353 19943 tablet_replica.cc:330] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: stopping tablet replica
I20260928 15:44:55.736672 19943 master.cc:552] Master@127.13.5.125:45387 shutdown complete.
I20260928 15:44:55.736790 19943 master_runner.cc:305] Clearing existing system tablet
I20260928 15:44:55.737277 19943 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:55.737396 19943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:55.740970 19943 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 15:44:55.742913 20046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:55.743353 19943 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:55.743644 19943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 15:44:55.743737 19943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
uuid: "ba69234c12ab4e909a9e6bfc2b2843fe"
format_stamp: "Formatted at 2026-09-28 15:44:55 on dist-test-slave-j968"
server_key: "c8e9a451a6e588790d5764d14aef9be2"
server_key_iv: "1cd00b17269988293581b1590577d810"
server_key_version: "encryptionkey@0"
I20260928 15:44:55.749292 19943 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:55.757313 19943 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 15:44:55.757532 19943 log.cc:1196] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 15:44:55.758044 19943 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Deleting consensus metadata
I20260928 15:44:55.758325 19943 master_runner.cc:315] Copying system tablet from 127.13.5.126:36553
I20260928 15:44:55.761313 19943 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: tablet copy: Beginning tablet copy session from remote peer at address 127.13.5.126:36553
I20260928 15:44:55.778187 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.762285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.13.5.126:36553)
0928 15:44:55.762499 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:55.762505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:55.763162 (+ 657us) server_negotiation.cc:407] Connection header received
0928 15:44:55.763396 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:55.763404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:55.763507 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:55.763623 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:55.764738 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.765985 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.767913 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.768583 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.768715 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.771098 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:55.771107 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:55.771112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:55.771164 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:55.774922 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.775794 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.775800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.775804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.776005 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:55.776489 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:55.776495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:55.776498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:55.776882 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:55.777030 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:55.777536 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 15:44:55.777794 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:44:55.778214 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:55.762226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:55830)
0928 15:44:55.762983 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:55.763009 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:55.763177 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:55.763675 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:55.763680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:55.764255 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:55.764569 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.764582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.766140 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.766145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.767706 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:55.767723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:55.768753 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:55.768759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:55.768964 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:55.769684 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:55.769710 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:55.770828 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:55.775088 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.775094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.775099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.775656 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.776125 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:55.776131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:55.776135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:55.776322 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:55.777049 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:55.777054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:55.777222 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 15:44:55.777402 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 15:44:55.777751 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":385,"threads_started":1}
I20260928 15:44:55.779567 19500 tablet_copy_service.cc:140] P e2722471774848d0923a854406fc2ba2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ba69234c12ab4e909a9e6bfc2b2843fe ({username='oryx', principal='oryx/127.13.5.125@KRBTEST.COM'} at 127.0.0.1:55830)
I20260928 15:44:55.779934 19500 tablet_copy_service.cc:161] P e2722471774848d0923a854406fc2ba2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ba69234c12ab4e909a9e6bfc2b2843fe at {username='oryx', principal='oryx/127.13.5.125@KRBTEST.COM'} at 127.0.0.1:55830: session id = ba69234c12ab4e909a9e6bfc2b2843fe-00000000000000000000000000000000
I20260928 15:44:55.784607 19500 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:44:55.787228 19943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 15:44:55.794397 19943 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: tablet copy: Starting download of 0 data blocks...
I20260928 15:44:55.794744 19943 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: tablet copy: Starting download of 1 WAL segments...
I20260928 15:44:55.798002 19943 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:44:55.802670 19500 tablet_copy_service.cc:342] P e2722471774848d0923a854406fc2ba2: Request end of tablet copy session ba69234c12ab4e909a9e6bfc2b2843fe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.5.125@KRBTEST.COM'} at 127.0.0.1:55830
I20260928 15:44:55.802927 19500 tablet_copy_service.cc:433] P e2722471774848d0923a854406fc2ba2: ending tablet copy session ba69234c12ab4e909a9e6bfc2b2843fe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ba69234c12ab4e909a9e6bfc2b2843fe
I20260928 15:44:55.816239 19943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:55.819123 20053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:55.819998 20054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:55.821776 20056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:55.824717 19943 server_base.cc:1034] running on GCE node
I20260928 15:44:55.825173 19943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:55.825803 19943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:55.827034 19943 hybrid_clock.cc:648] HybridClock initialized: now 1790610295826971 us; error 33 us; skew 500 ppm
I20260928 15:44:55.827282 19943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:55 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610295, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:55.829216 19943 init.cc:377] Logged in from keytab as oryx/127.13.5.125@KRBTEST.COM (short username oryx)
I20260928 15:44:55.830487 19943 webserver.cc:460] Webserver started at http://127.13.5.125:42573/ using document root <none> and password file <none>
I20260928 15:44:55.830819 19943 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:55.830916 19943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:55.833343 19943 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 15:44:55.835068 20063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:55.835501 19943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:44:55.835645 19943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
uuid: "ba69234c12ab4e909a9e6bfc2b2843fe"
format_stamp: "Formatted at 2026-09-28 15:44:55 on dist-test-slave-j968"
server_key: "c8e9a451a6e588790d5764d14aef9be2"
server_key_iv: "1cd00b17269988293581b1590577d810"
server_key_version: "encryptionkey@0"
I20260928 15:44:55.835813 19943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:55.846877 19943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:55.850389 19943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:55.877015 19943 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.125:45387
I20260928 15:44:55.877028 20114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.125:45387 every 8 connection(s)
I20260928 15:44:55.878049 19943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
I20260928 15:44:55.885690 20115 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:44:55.891443 20115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Bootstrap starting.
I20260928 15:44:55.907270 20115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:44:55.907770 20115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: Bootstrap complete.
I20260928 15:44:55.908751 20115 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } }
I20260928 15:44:55.909050 20115 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba69234c12ab4e909a9e6bfc2b2843fe, State: Initialized, Role: LEARNER
I20260928 15:44:55.909297 20115 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } }
I20260928 15:44:55.910122 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } }
I20260928 15:44:55.910449 20115 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:55.910492 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: This master's current role is: LEARNER
I20260928 15:44:55.917989 20129 catalog_manager.cc:1259] Loaded cluster ID: 61c2b26ae621471b90884dc025292186
I20260928 15:44:55.918107 20129 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: loading cluster ID for follower catalog manager: success
I20260928 15:44:55.921000 20129 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: acquiring CA information for follower catalog manager: success
I20260928 15:44:55.922917 20129 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:44:55.987382 19937 heartbeater.cc:503] Master 127.13.5.126:36553 was elected leader, sending a full tablet report...
W20260928 15:44:56.110718 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:56.111096 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:56.111271 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:56.119163 20044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:44:56.119306 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:56.119406 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:56.119506 20044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:44:56.119601 20044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:44:56.129952 20044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:36553,127.13.5.125:45387,127.13.5.124:33123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.124.keytab
--principal=oryx/127.13.5.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.124:33123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.5.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:56.131390 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:56.133057 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:56.147801 20137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:56.147786 20135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:56.147949 20134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:56.148718 20044 server_base.cc:1034] running on GCE node
I20260928 15:44:56.149516 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:56.150969 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:56.152206 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790610296152160 us; error 43 us; skew 500 ppm
I20260928 15:44:56.152565 20044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:56 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610296, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:56.156041 20044 init.cc:377] Logged in from keytab as oryx/127.13.5.124@KRBTEST.COM (short username oryx)
I20260928 15:44:56.157740 20044 webserver.cc:460] Webserver started at http://127.13.5.124:35809/ using document root <none> and password file <none>
I20260928 15:44:56.158473 20044 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:56.158638 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:56.158960 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:44:56.162438 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data/instance:
uuid: "fdfb7623849d4bb1859a7f7f15d2ba90"
format_stamp: "Formatted at 2026-09-28 15:44:56 on dist-test-slave-j968"
server_key: "762559b94b3214b30b201f2ad31ad358"
server_key_iv: "b53ce0d50bb2e6bae76d1f68b72cc9f2"
server_key_version: "encryptionkey@0"
I20260928 15:44:56.163348 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal/instance:
uuid: "fdfb7623849d4bb1859a7f7f15d2ba90"
format_stamp: "Formatted at 2026-09-28 15:44:56 on dist-test-slave-j968"
server_key: "762559b94b3214b30b201f2ad31ad358"
server_key_iv: "b53ce0d50bb2e6bae76d1f68b72cc9f2"
server_key_version: "encryptionkey@0"
I20260928 15:44:56.169214 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:44:56.173266 20144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:56.174862 20044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260928 15:44:56.175089 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
uuid: "fdfb7623849d4bb1859a7f7f15d2ba90"
format_stamp: "Formatted at 2026-09-28 15:44:56 on dist-test-slave-j968"
server_key: "762559b94b3214b30b201f2ad31ad358"
server_key_iv: "b53ce0d50bb2e6bae76d1f68b72cc9f2"
server_key_version: "encryptionkey@0"
I20260928 15:44:56.175320 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:56.194199 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:56.199172 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:56.199470 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:56.221462 20044 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.124:33123
I20260928 15:44:56.221532 20198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.124:33123 every 8 connection(s)
I20260928 15:44:56.223171 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
I20260928 15:44:56.223182 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.215167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.13.5.125:45387)
0928 15:44:56.216027 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.216035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.216058 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:44:56.216145 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.216153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.216270 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.216421 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:56.216860 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.218252 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.220927 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.222166 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.222287 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.222485 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.222789 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.222908 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":434,"threads_started":1}
I20260928 15:44:56.223222 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.214918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.0.0.1:58848)
0928 15:44:56.215458 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.215497 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.215710 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.216480 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.216484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.216509 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:56.216679 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.216686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.218450 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.218482 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.220730 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.220751 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.222361 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.222367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.222544 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.222612 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.222731 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.222875 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":218,"threads_started":1}
I20260928 15:44:56.224844 20090 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 15:44:56.225093 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 20044
I20260928 15:44:56.226255 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2722471774848d0923a854406fc2ba2. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } }
I20260928 15:44:56.226537 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: This master's current role is: LEARNER
I20260928 15:44:56.228802 20199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:56.237748 20199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:44:56 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.124@KRBTEST.COM
Sep 28 15:44:56 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610296, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.124@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:44:56.250178 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.227685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.13.5.124:33123)
0928 15:44:56.228801 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.228813 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.228856 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:44:56.230474 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.230496 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.230872 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.231187 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.233017 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.234378 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.236302 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.237082 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.237284 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.242758 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.242766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.242780 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.242820 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.245940 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.246686 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.246692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.246699 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.246867 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.247282 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.247292 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.247294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.247834 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.248219 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.248689 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.249091 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":594,"threads_started":1}
I20260928 15:44:56.255993 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.241340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.13.5.126:36553)
0928 15:44:56.241648 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.241658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.242319 (+ 661us) server_negotiation.cc:407] Connection header received
0928 15:44:56.242567 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.242571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.242637 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.242752 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.243967 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.244714 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.246150 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.246567 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.246699 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.249979 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.249984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.249988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.250028 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.253271 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.253936 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.253943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.253949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.254105 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.254524 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.254531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.254534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.254806 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.254961 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.255412 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.255620 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:44:56.256057 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.241257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:55842)
0928 15:44:56.242131 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.242193 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.242393 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.242805 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.242813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.243491 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:56.243791 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.243800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.244862 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.244867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.245997 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.246016 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.246764 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.246773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.246939 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.247547 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:56.247565 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:56.249794 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:56.253398 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.253410 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.253417 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.253813 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.254220 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.254225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.254228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.254401 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.254932 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:56.254940 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:56.255115 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.255289 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.255619 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":404,"threads_started":1}
I20260928 15:44:56.257390 20199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:44:56 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610296, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.124@KRBTEST.COM for oryx/127.13.5.125@KRBTEST.COM
I20260928 15:44:56.270653 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.258189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.13.5.125:45387)
0928 15:44:56.258380 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.258386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.258485 (+ 99us) server_negotiation.cc:407] Connection header received
0928 15:44:56.258673 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.258678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.258740 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.258841 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.259561 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.260355 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.261583 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.262004 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.262141 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.265372 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.265377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.265380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.265413 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.268141 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.268878 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.268881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.268883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.269032 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.269451 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.269455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.269457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.269736 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.269846 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.270210 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.270390 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:44:56.270742 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.258184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.0.0.1:58864)
0928 15:44:56.258384 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.258403 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.258544 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.258860 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.258865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.259200 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:56.259324 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.259331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.260496 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.260501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.261445 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.261456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.262154 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.262159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.262385 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.262993 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:56.263009 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:56.265179 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:56.268268 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.268273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.268278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.268770 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.269143 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.269147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.269151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.269320 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.269843 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:56.269848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:56.269996 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.270134 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.270415 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 15:44:56.271699 20199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:44:56 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610296, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.124@KRBTEST.COM for oryx/127.13.5.124@KRBTEST.COM
I20260928 15:44:56.285346 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.272518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33123 (local address 127.0.0.1:40180)
0928 15:44:56.272744 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.272762 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.272853 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.273109 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.273112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.273476 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:56.273626 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.273633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.274995 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.274999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.275856 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.275942 (+ 86us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.276936 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.276941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.277117 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.277720 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:56.277743 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:56.280155 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:56.283191 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.283195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.283197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.283555 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.283932 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.283941 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.283944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.284093 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.284586 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:56.284589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:56.284712 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.284811 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.285001 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 15:44:56.285508 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.272691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.13.5.124:33123)
0928 15:44:56.272858 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.272864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.272882 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:44:56.272925 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.272930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.272993 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.273081 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.273755 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.274864 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.276083 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.276789 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.276903 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.280339 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.280344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.280347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.280444 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.283070 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.283656 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.283660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.283662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.283780 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.284189 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.284193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.284195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.284470 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.284577 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.284912 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.285120 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":55}
I20260928 15:44:56.290735 20199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Bootstrap starting.
I20260928 15:44:56.294313 20199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:56.295449 20199 log.cc:824] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:56.298477 20199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: No bootstrap required, opened a new log
I20260928 15:44:56.306562 20199 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } }
I20260928 15:44:56.306929 20199 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:56.307024 20199 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfb7623849d4bb1859a7f7f15d2ba90, State: Initialized, Role: FOLLOWER
I20260928 15:44:56.307643 20199 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } }
I20260928 15:44:56.308626 20205 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } } }
I20260928 15:44:56.308949 20205 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:44:56.309747 20199 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:44:56.320186 20216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:44:56.320350 20216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:44:56.324468 20044 master_runner.cc:428] Detected that this master fdfb7623849d4bb1859a7f7f15d2ba90 is joining an existing cluster
I20260928 15:44:56.324601 20044 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.5.124:33123
I20260928 15:44:56.328141 20205 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:56.328429 20205 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } }
I20260928 15:44:56.330206 20205 leader_election.cc:288] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2722471774848d0923a854406fc2ba2 (127.13.5.126:36553), ba69234c12ab4e909a9e6bfc2b2843fe (127.13.5.125:45387)
I20260928 15:44:56.330917 19490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2722471774848d0923a854406fc2ba2" is_pre_election: true
I20260928 15:44:56.331302 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" is_pre_election: true
I20260928 15:44:56.332304 20148 leader_election.cc:302] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfb7623849d4bb1859a7f7f15d2ba90; no voters: ba69234c12ab4e909a9e6bfc2b2843fe, e2722471774848d0923a854406fc2ba2
I20260928 15:44:56.333706 20205 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:44:56.352299 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.329518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.13.5.125:45387)
0928 15:44:56.329895 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.329900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.335019 (+ 5119us) server_negotiation.cc:407] Connection header received
0928 15:44:56.335287 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.335294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.335383 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.335489 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.337079 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.338210 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.340291 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.340963 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.341066 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.345127 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.345134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.345139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.345182 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.349054 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.349817 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.349823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.349826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.349993 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.350486 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.350491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.350494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.350830 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.350976 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.351610 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.351885 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260928 15:44:56.353884 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.329574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.0.0.1:58868)
0928 15:44:56.334877 (+ 5303us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.334907 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.335120 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.335697 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.335702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.336471 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:56.336876 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.336899 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.338408 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.338413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.340132 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.340146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.342480 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.342487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.342773 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.343558 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:56.343584 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:56.344909 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:56.349205 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.349211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.349215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.349681 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.350141 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.350146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.350150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.350349 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.351055 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:56.351060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:56.351306 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.352570 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.352926 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5046,"mutex_wait_us":74,"thread_start_us":4863,"threads_started":1}
I20260928 15:44:56.356235 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.329202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.13.5.126:36553)
0928 15:44:56.329733 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.329740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.336028 (+ 6288us) server_negotiation.cc:407] Connection header received
0928 15:44:56.336365 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.336378 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.336480 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.336564 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.338527 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.340070 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.343053 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.344016 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.344142 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.347742 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.347751 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.347759 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.347813 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.352181 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.352923 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.352929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.352932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.353068 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.353898 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.353906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.353911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.354747 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.354879 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.355593 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.355844 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389}
I20260928 15:44:56.357409 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 19541
I20260928 15:44:56.357577 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.332292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:55850)
0928 15:44:56.335923 (+ 3631us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.335946 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.336178 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.336815 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.336826 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.337863 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:56.338305 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.338326 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.340343 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.340350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.342806 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.342828 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.344517 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.344526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.344843 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.345699 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:56.345732 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:56.347461 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:56.352290 (+ 4829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.352297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.352302 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.352783 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.353204 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.353209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.353212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.353739 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.354977 (+ 1238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:56.354982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:56.355206 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.356742 (+ 1536us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.357114 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3430,"mutex_wait_us":11,"thread_start_us":3197,"threads_started":1}
I20260928 15:44:56.371539 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.336261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33123 (local address 127.0.0.1:40188)
0928 15:44:56.352335 (+ 16074us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.352361 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.352541 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.352968 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.352972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.354016 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:56.354331 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.354343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.356041 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.356046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.359489 (+ 3443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.359509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.361086 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.361095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.361345 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.362197 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:56.362234 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:56.364220 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:56.368578 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.368585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.368591 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.369109 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.369547 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.369552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.369555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.369706 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.370345 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:56.370361 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:56.370556 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.370689 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.371002 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15924,"mutex_wait_us":3,"thread_start_us":15770,"threads_started":1}
I20260928 15:44:56.372571 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.336097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.13.5.124:33123)
0928 15:44:56.336286 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.336293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.352448 (+ 16155us) server_negotiation.cc:407] Connection header received
0928 15:44:56.352657 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.352662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.352725 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.352782 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.354490 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.355565 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.359693 (+ 4128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.360578 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.360703 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.364423 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.364430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.364435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.364471 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.368440 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.369250 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.369255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.369259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.369436 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.369829 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.369834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.369838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.370211 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.371810 (+ 1599us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.371982 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.372198 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 15:44:56 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.65@KRBTEST.COM
Sep 28 15:44:56 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610293, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.126@KRBTEST.COM for oryx/127.13.5.124@KRBTEST.COM
I20260928 15:44:56.390514 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.363694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33123 (local address 127.0.0.1:40196)
0928 15:44:56.363882 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.363909 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.364019 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.373159 (+ 9140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.373168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.373670 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:56.373869 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.373881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.375449 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.375456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.376969 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.376989 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.378185 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.378192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.378417 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.379140 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:56.379176 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:56.382948 (+ 3772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:56.387296 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.387302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.387306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.387957 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.388395 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.388400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.388404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.388882 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.389540 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:56.389545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:56.389724 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.389844 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.390141 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 15:44:56.390787 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.358814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.13.5.65:33255)
0928 15:44:56.364930 (+ 6116us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.364943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.364983 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:44:56.365069 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.365082 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.365453 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.367983 (+ 2530us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.368331 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.370476 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.373274 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.374285 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.381175 (+ 6890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.382902 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.382920 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.382935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.382969 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.387295 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.388029 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.388039 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.388042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.388174 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.388644 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.388650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.388654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.389158 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.389479 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.389880 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.390171 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5875,"thread_start_us":6989,"threads_started":1}
I20260928 15:44:56.391629 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.364226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.13.5.124:33123)
0928 15:44:56.372796 (+ 8570us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.372801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.372818 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:44:56.372909 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.372914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.372996 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.373108 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.374046 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.375260 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.377185 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.377971 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.378734 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.383153 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.383162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.383167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.383206 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.387129 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.388110 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.388115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.388119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.388275 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.389012 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.389017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.389021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.389378 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.390906 (+ 1528us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.391051 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.391284 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8472,"thread_start_us":17429,"threads_started":1}
I20260928 15:44:56.392419 19664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:56.393041 19479 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.5.124:33123: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90"
member_type: NON_VOTER
last_known_addr {
host: "127.13.5.124"
port: 33123
}
attrs {
promote: true
}
}
dest_uuid: "e2722471774848d0923a854406fc2ba2"
cas_config_opid_index: 5
I20260928 15:44:56.394060 19479 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } }
I20260928 15:44:56.396134 20178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fdfb7623849d4bb1859a7f7f15d2ba90 (127.13.5.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.401376 20090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 LEARNER]: Refusing update from remote peer e2722471774848d0923a854406fc2ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 15:44:56.403461 20171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:56.404596 20228 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 15:44:56.409834 20178 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.5.124:33123
I20260928 15:44:56.409996 20171 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Refusing update from remote peer e2722471774848d0923a854406fc2ba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 15:44:56.414271 20044 master.cc:537] Master@127.13.5.124:33123 shutting down...
I20260928 15:44:56.414301 20178 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.415119 20178 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:56.421087 20229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:56.425877 20090 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fdfb7623849d4bb1859a7f7f15d2ba90 (127.13.5.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.434041 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: NON_VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: true } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.434310 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: This master's current role is: LEARNER
I20260928 15:44:56.435231 20229 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: attempting to promote NON_VOTER ba69234c12ab4e909a9e6bfc2b2843fe to VOTER
I20260928 15:44:56.436260 20229 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } }
I20260928 15:44:56.439213 20090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 LEARNER]: Refusing update from remote peer e2722471774848d0923a854406fc2ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 15:44:56.439811 20229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 15:44:56.442219 20231 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ba69234c12ab4e909a9e6bfc2b2843fe (127.13.5.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.445377 20231 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.445662 20231 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:56.445971 20229 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.446241 20229 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:56.446211 20090 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ba69234c12ab4e909a9e6bfc2b2843fe (127.13.5.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.449138 20118 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.449378 20118 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:44:56.453413 20150 tablet.cc:2369] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:56.455618 19466 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 -> Peer fdfb7623849d4bb1859a7f7f15d2ba90 (127.13.5.124:33123): Couldn't send request to peer fdfb7623849d4bb1859a7f7f15d2ba90. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 15:44:56.455621 20044 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:56.456085 20044 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:56.456285 20044 tablet_replica.cc:330] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: stopping tablet replica
I20260928 15:44:56.463536 20044 master.cc:552] Master@127.13.5.124:33123 shutdown complete.
I20260928 15:44:56.463647 20044 master_runner.cc:305] Clearing existing system tablet
I20260928 15:44:56.464169 20044 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:56.464299 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:56.466514 20044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:44:56.468083 20235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:56.468400 20044 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:56.468708 20044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:44:56.468822 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
uuid: "fdfb7623849d4bb1859a7f7f15d2ba90"
format_stamp: "Formatted at 2026-09-28 15:44:56 on dist-test-slave-j968"
server_key: "762559b94b3214b30b201f2ad31ad358"
server_key_iv: "b53ce0d50bb2e6bae76d1f68b72cc9f2"
server_key_version: "encryptionkey@0"
I20260928 15:44:56.472995 20044 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:56.477931 20044 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 15:44:56.478127 20044 log.cc:1196] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 15:44:56.478627 20044 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Deleting consensus metadata
I20260928 15:44:56.478886 20044 master_runner.cc:315] Copying system tablet from 127.13.5.126:36553
I20260928 15:44:56.481644 20044 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: tablet copy: Beginning tablet copy session from remote peer at address 127.13.5.126:36553
I20260928 15:44:56.495365 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.482444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.13.5.126:36553)
0928 15:44:56.482763 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.482772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.483438 (+ 666us) server_negotiation.cc:407] Connection header received
0928 15:44:56.483650 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.483656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.483735 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.483843 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:56.484757 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.485482 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.486779 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.487208 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.487361 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.489267 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:56.489273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:56.489278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:56.489318 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:56.492201 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.492929 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.492944 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.492946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.493082 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:56.493588 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:56.493592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:56.493594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:56.494059 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:56.494235 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.494640 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 15:44:56.494949 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 15:44:56.495451 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.482424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:55864)
0928 15:44:56.483302 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.483326 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.483508 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.483892 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.483898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.484376 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:56.484598 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.484611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.485665 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.485670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.486623 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.486635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.487370 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.487375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.487578 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.488251 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:56.488269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:56.489071 (+ 802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:56.492340 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.492345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.492350 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.492794 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.493194 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:56.493199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:56.493203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:56.493363 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:56.494180 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:56.494186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:56.494366 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.494519 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.494949 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":569,"threads_started":1}
I20260928 15:44:56.496345 19500 tablet_copy_service.cc:140] P e2722471774848d0923a854406fc2ba2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fdfb7623849d4bb1859a7f7f15d2ba90 ({username='oryx', principal='oryx/127.13.5.124@KRBTEST.COM'} at 127.0.0.1:55864)
I20260928 15:44:56.496548 19500 tablet_copy_service.cc:161] P e2722471774848d0923a854406fc2ba2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fdfb7623849d4bb1859a7f7f15d2ba90 at {username='oryx', principal='oryx/127.13.5.124@KRBTEST.COM'} at 127.0.0.1:55864: session id = fdfb7623849d4bb1859a7f7f15d2ba90-00000000000000000000000000000000
I20260928 15:44:56.499166 19500 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:44:56.500968 20044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 15:44:56.506323 20044 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: tablet copy: Starting download of 0 data blocks...
I20260928 15:44:56.506668 20044 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: tablet copy: Starting download of 1 WAL segments...
I20260928 15:44:56.509008 20044 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:44:56.513267 19500 tablet_copy_service.cc:342] P e2722471774848d0923a854406fc2ba2: Request end of tablet copy session fdfb7623849d4bb1859a7f7f15d2ba90-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.5.124@KRBTEST.COM'} at 127.0.0.1:55864
I20260928 15:44:56.513432 19500 tablet_copy_service.cc:433] P e2722471774848d0923a854406fc2ba2: ending tablet copy session fdfb7623849d4bb1859a7f7f15d2ba90-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fdfb7623849d4bb1859a7f7f15d2ba90
I20260928 15:44:56.531477 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:56.534945 20242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:56.540484 20243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:56.547930 20044 server_base.cc:1034] running on GCE node
W20260928 15:44:56.548300 20245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:56.548760 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:56.549400 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:56.550596 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790610296550561 us; error 27 us; skew 500 ppm
I20260928 15:44:56.550835 20044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:56 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610296, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:56.552878 20044 init.cc:377] Logged in from keytab as oryx/127.13.5.124@KRBTEST.COM (short username oryx)
I20260928 15:44:56.560364 20044 webserver.cc:460] Webserver started at http://127.13.5.124:43243/ using document root <none> and password file <none>
I20260928 15:44:56.560719 20044 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:56.560844 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:56.563949 20044 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 15:44:56.565801 20252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:56.566277 20044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260928 15:44:56.566434 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
uuid: "fdfb7623849d4bb1859a7f7f15d2ba90"
format_stamp: "Formatted at 2026-09-28 15:44:56 on dist-test-slave-j968"
server_key: "762559b94b3214b30b201f2ad31ad358"
server_key_iv: "b53ce0d50bb2e6bae76d1f68b72cc9f2"
server_key_version: "encryptionkey@0"
I20260928 15:44:56.566668 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:56.572715 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 19541
I20260928 15:44:56.598582 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:33255
--local_ip_for_outbound_sockets=127.13.5.65
--tserver_master_addrs=127.13.5.126:36553
--webserver_port=34209
--webserver_interface=127.13.5.65
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab
--principal=oryx/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387,127.13.5.124:33123 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:44:56.600350 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:56.604092 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:56.636591 20044 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.124:33123
I20260928 15:44:56.636539 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.124:33123 every 8 connection(s)
I20260928 15:44:56.637633 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
I20260928 15:44:56.644819 20305 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:44:56.650362 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Bootstrap starting.
I20260928 15:44:56.669589 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:44:56.670085 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Bootstrap complete.
I20260928 15:44:56.670953 20305 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } }
I20260928 15:44:56.671208 20305 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfb7623849d4bb1859a7f7f15d2ba90, State: Initialized, Role: LEARNER
I20260928 15:44:56.671451 20305 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } }
I20260928 15:44:56.678066 20305 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:44:56.677682 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:56.678355 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:44:56.687490 20319 catalog_manager.cc:1259] Loaded cluster ID: 61c2b26ae621471b90884dc025292186
I20260928 15:44:56.687614 20319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: loading cluster ID for follower catalog manager: success
I20260928 15:44:56.690078 20319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: acquiring CA information for follower catalog manager: success
I20260928 15:44:56.691838 20319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 15:44:56.997617 20259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:56.997982 20259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:56.998160 20259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 15:44:57.000205 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.992704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33123 (local address 127.0.0.1:40200)
0928 15:44:56.993602 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:56.993618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:56.993772 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:56.994309 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:56.994313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:56.994336 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:56.994572 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.994581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.996487 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.996491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.997708 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:56.997721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.999338 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.999343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:56.999652 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.999698 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 15:44:56.999786 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 15:44:56.999985 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":624,"threads_started":1}
I20260928 15:44:57.000391 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:56.992844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.13.5.124:33123)
0928 15:44:56.993406 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:56.993412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:56.993744 (+ 332us) server_negotiation.cc:407] Connection header received
0928 15:44:56.994046 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:56.994053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:56.994151 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:56.994281 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:56.994708 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.996323 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.997871 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:56.999141 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:56.999274 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:56.999504 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:56.999839 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.000011 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":257,"threads_started":1}
I20260928 15:44:57.001405 20280 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 15:44:57.002002 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2722471774848d0923a854406fc2ba2. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:57.002315 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: This master's current role is: LEARNER
W20260928 15:44:57.007483 20259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:57.007589 20259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:57.007704 20259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:44:57.018989 20259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab
--principal=oryx/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:33255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=34209
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387,127.13.5.124:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:57.020501 20259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:57.021888 20259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:57.036871 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:57.036906 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:57.036895 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:57.038007 20259 server_base.cc:1034] running on GCE node
I20260928 15:44:57.038652 20259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:57.039629 20259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:57.040869 20259 hybrid_clock.cc:648] HybridClock initialized: now 1790610297040825 us; error 49 us; skew 500 ppm
I20260928 15:44:57.041230 20259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610297, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:57.044595 20259 init.cc:377] Logged in from keytab as oryx/127.13.5.65@KRBTEST.COM (short username oryx)
I20260928 15:44:57.046218 20259 webserver.cc:460] Webserver started at http://127.13.5.65:34209/ using document root <none> and password file <none>
I20260928 15:44:57.046898 20259 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:57.047014 20259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:57.052492 20259 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:44:57.055223 20336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:57.056799 20259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:57.056974 20259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "5b75b8b58bd4425499d56d72b7158364"
format_stamp: "Formatted at 2026-09-28 15:44:53 on dist-test-slave-j968"
server_key: "d60df972dd57241a4747e61fdadb155f"
server_key_iv: "950996194171f5e6e81701c841668511"
server_key_version: "encryptionkey@0"
I20260928 15:44:57.057976 20259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:57.078804 20259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:57.083268 20259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:57.083570 20259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:57.084664 20259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:57.086136 20259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:57.086256 20259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:57.086367 20259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:57.086462 20259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:57.140661 20259 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:33255
I20260928 15:44:57.140908 20448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:33255 every 8 connection(s)
I20260928 15:44:57.142561 20259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:44:57.147439 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 20259
I20260928 15:44:57.147957 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 19675
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.66@KRBTEST.COM
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610297, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610297, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.125@KRBTEST.COM
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610297, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.124@KRBTEST.COM
I20260928 15:44:57.199112 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.149593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.13.5.66:43093)
0928 15:44:57.167282 (+ 17689us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:57.167291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:57.167322 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:44:57.167394 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:57.167405 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:57.167855 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:57.168143 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:57.169837 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.171780 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.173853 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.174822 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.174962 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.185125 (+ 10163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:57.185156 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:57.185184 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:57.185241 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:57.192221 (+ 6980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.193916 (+ 1695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.193929 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.193934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.194323 (+ 389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.194984 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.194990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.194994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.195709 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:57.196137 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:57.198007 (+ 1870us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.198379 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17378,"thread_start_us":17030,"threads_started":1}
I20260928 15:44:57.200860 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.150995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:48815 (local address 127.13.5.125:45387)
0928 15:44:57.164837 (+ 13842us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:57.164846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:57.170553 (+ 5707us) server_negotiation.cc:407] Connection header received
0928 15:44:57.172904 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:57.172910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:57.173020 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:57.173122 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:57.176218 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.177367 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.180553 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.181627 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.181735 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.189265 (+ 7530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:57.189273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:57.189278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:57.189323 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:57.195845 (+ 6522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.197222 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.197237 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.197241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.197477 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.198139 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.198143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.198146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.198624 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:57.198781 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:57.200086 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.200312 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13639,"thread_start_us":13515,"threads_started":1}
I20260928 15:44:57.200872 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.170474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:39761 (local address 127.13.5.126:36553)
0928 15:44:57.171258 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:57.171264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:57.172199 (+ 935us) server_negotiation.cc:407] Connection header received
0928 15:44:57.172798 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:57.172804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:57.172884 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:57.172972 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:57.176252 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.177759 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.181000 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.182106 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.182210 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.188693 (+ 6483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:57.188702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:57.188709 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:57.188770 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:57.195365 (+ 6595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.196957 (+ 1592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.196962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.196965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.197168 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.197924 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.197932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.197938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.198470 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:57.198639 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:57.199890 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.200155 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":539,"threads_started":1}
I20260928 15:44:57.201548 19798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:57.203894 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.150213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.13.5.65:48815)
0928 15:44:57.170184 (+ 19971us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:57.170407 (+ 223us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:57.172454 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:57.173464 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:57.173478 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:57.174648 (+ 1170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:57.176002 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.176031 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.177580 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.177585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.180374 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.180389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.182338 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.182357 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.182851 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.184120 (+ 1269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:57.184162 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:57.188828 (+ 4666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:57.196103 (+ 7275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.196108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.196111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.197031 (+ 920us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.197698 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.197702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.197705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.197958 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.198877 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:57.198882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:57.199350 (+ 468us) client_negotiation.cc:769] Sending connection context
0928 15:44:57.201469 (+ 2119us) client_negotiation.cc:240] Negotiation successful
0928 15:44:57.201892 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19755,"mutex_wait_us":409,"spinlock_wait_cycles":36224,"thread_start_us":15207,"threads_started":1}
I20260928 15:44:57.204071 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.147116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:42839 (local address 127.13.5.124:33123)
0928 15:44:57.147383 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:57.147391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:57.166141 (+ 18750us) server_negotiation.cc:407] Connection header received
0928 15:44:57.172806 (+ 6665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:57.172812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:57.172904 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:57.172977 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:57.176218 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.177345 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.180589 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.181647 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.181734 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.191438 (+ 9704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:57.191446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:57.191453 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:57.191518 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:57.197036 (+ 5518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.198078 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.198086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.198092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.198342 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.198905 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.198909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.198912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.199291 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:57.199484 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:57.200109 (+ 625us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.200410 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:44:57.204690 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.170365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.65:39761)
0928 15:44:57.172081 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:57.172100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:57.172469 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:57.173465 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:57.173489 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:57.174650 (+ 1161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:57.176050 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.176071 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.178036 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.178042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.180771 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.180788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.182735 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.182744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.183127 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.184240 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:57.184267 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:57.188242 (+ 3975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:57.195618 (+ 7376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.195638 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.195659 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.196768 (+ 1109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.197342 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.197346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.197349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.197776 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.198758 (+ 982us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:57.198767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:57.199360 (+ 593us) client_negotiation.cc:769] Sending connection context
0928 15:44:57.201150 (+ 1790us) client_negotiation.cc:240] Negotiation successful
0928 15:44:57.202117 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1577,"mutex_wait_us":450,"spinlock_wait_cycles":9728,"thread_start_us":1394,"threads_started":1}
I20260928 15:44:57.205574 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.148644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33123 (local address 127.13.5.65:42839)
0928 15:44:57.165948 (+ 17304us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:57.166012 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:57.172454 (+ 6442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:57.173465 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:57.173478 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:57.174648 (+ 1170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:57.176001 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.176031 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.177580 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.177585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.180374 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.180389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.182371 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.182377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.182847 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.184121 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:57.184163 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:57.191080 (+ 6917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:57.197214 (+ 6134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.197221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.197227 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.197891 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.198500 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.198506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.198511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.198776 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.199591 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:57.199595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:57.199723 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 15:44:57.204308 (+ 4585us) client_negotiation.cc:240] Negotiation successful
0928 15:44:57.204806 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16914,"mutex_wait_us":508,"spinlock_wait_cycles":60928,"thread_start_us":18816,"threads_started":1}
I20260928 15:44:57.207453 20451 heartbeater.cc:347] Connected to a master server at 127.13.5.125:45387
I20260928 15:44:57.207619 20450 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36553
I20260928 15:44:57.207693 20449 heartbeater.cc:347] Connected to a master server at 127.13.5.124:33123
I20260928 15:44:57.208003 20449 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:57.208021 20451 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:57.208003 20450 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:57.209987 20449 heartbeater.cc:511] Master 127.13.5.124:33123 requested a full tablet report, sending...
I20260928 15:44:57.210290 20451 heartbeater.cc:511] Master 127.13.5.125:45387 requested a full tablet report, sending...
I20260928 15:44:57.211846 20450 heartbeater.cc:511] Master 127.13.5.126:36553 requested a full tablet report, sending...
I20260928 15:44:57.212917 20080 ts_manager.cc:194] Registered new tserver with Master: 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255)
I20260928 15:44:57.213233 20270 ts_manager.cc:194] Registered new tserver with Master: 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255)
I20260928 15:44:57.214411 19479 ts_manager.cc:194] Re-registered known tserver with Master: 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255)
I20260928 15:44:57.215400 19479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:39761
I20260928 15:44:57.295710 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 19675
I20260928 15:44:57.314469 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:43093
--local_ip_for_outbound_sockets=127.13.5.66
--tserver_master_addrs=127.13.5.126:36553
--webserver_port=42023
--webserver_interface=127.13.5.66
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab
--principal=oryx/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387,127.13.5.124:33123 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:44:57.392508 20229 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: NON_VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: true } } }
I20260928 15:44:57.392925 20229 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:57.395455 20462 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: attempting to promote NON_VOTER fdfb7623849d4bb1859a7f7f15d2ba90 to VOTER
I20260928 15:44:57.396377 20462 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } }
I20260928 15:44:57.398518 20280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 LEARNER]: Refusing update from remote peer e2722471774848d0923a854406fc2ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 15:44:57.398725 20090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Refusing update from remote peer e2722471774848d0923a854406fc2ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 15:44:57.399147 20462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 15:44:57.399575 20229 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 15:44:57.402208 20229 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfb7623849d4bb1859a7f7f15d2ba90 (127.13.5.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:44:57.403074 20090 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfb7623849d4bb1859a7f7f15d2ba90 (127.13.5.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:44:57.405318 20229 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:44:57.405558 20229 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:57.405987 20229 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:44:57.406343 20463 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:44:57.406667 20463 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:57.406862 20229 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:44:57.406893 20280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfb7623849d4bb1859a7f7f15d2ba90 (127.13.5.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:44:57.408382 20466 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:44:57.408759 20466 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:44:57.410789 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2722471774848d0923a854406fc2ba2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:44:57.411089 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:44:57.663841 20461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:57.664235 20461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:57.664403 20461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:57.672046 20461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:57.672199 20461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:57.672394 20461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:44:57.683145 20461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab
--principal=oryx/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:43093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=42023
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387,127.13.5.124:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:57.684593 20461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:57.685971 20461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:57.700060 20471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:57.700165 20472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:57.700183 20474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:57.700830 20461 server_base.cc:1034] running on GCE node
I20260928 15:44:57.701681 20461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:57.702706 20461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:57.704007 20461 hybrid_clock.cc:648] HybridClock initialized: now 1790610297703961 us; error 35 us; skew 500 ppm
I20260928 15:44:57.704363 20461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610297, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:57.707809 20461 init.cc:377] Logged in from keytab as oryx/127.13.5.66@KRBTEST.COM (short username oryx)
I20260928 15:44:57.709496 20461 webserver.cc:460] Webserver started at http://127.13.5.66:42023/ using document root <none> and password file <none>
I20260928 15:44:57.710278 20461 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:57.710450 20461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:57.716634 20461 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:44:57.719563 20481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:57.721061 20461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:44:57.721251 20461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "dfbdcc4f879041f8953c4b203a208781"
format_stamp: "Formatted at 2026-09-28 15:44:54 on dist-test-slave-j968"
server_key: "1bb67e0c2dc00c3a73570bbde8db26d7"
server_key_iv: "a431f9ee3564c6e1e13593d1a4476cbc"
server_key_version: "encryptionkey@0"
I20260928 15:44:57.722291 20461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:57.737946 20461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:57.742043 20461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:57.742380 20461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:57.743429 20461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:57.744964 20461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:57.745150 20461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:57.745314 20461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:57.745532 20461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:57.780351 20461 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43093
I20260928 15:44:57.780440 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43093 every 8 connection(s)
I20260928 15:44:57.782114 20461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:44:57.786892 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 20461
I20260928 15:44:57.787240 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 19809
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610297, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.124@KRBTEST.COM
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.67@KRBTEST.COM
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610297, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:44:57.828064 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.785862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33123 (local address 127.13.5.66:57305)
0928 15:44:57.791406 (+ 5544us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:57.791454 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:57.795802 (+ 4348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:57.796830 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:57.796846 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:57.797810 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:57.798813 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.798844 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.801788 (+ 2944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.801798 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.805464 (+ 3666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.805487 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.808184 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.808194 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.808940 (+ 746us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.810352 (+ 1412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:57.810416 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:57.815182 (+ 4766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:57.821494 (+ 6312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.821509 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.821530 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.822452 (+ 922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.823134 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.823144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.823148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.823656 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.824401 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:57.824410 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:57.825066 (+ 656us) client_negotiation.cc:769] Sending connection context
0928 15:44:57.825554 (+ 488us) client_negotiation.cc:240] Negotiation successful
0928 15:44:57.826238 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5226,"thread_start_us":4966,"threads_started":1}
I20260928 15:44:57.829464 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.785255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:57305 (local address 127.13.5.124:33123)
0928 15:44:57.791030 (+ 5775us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:57.791040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:57.791583 (+ 543us) server_negotiation.cc:407] Connection header received
0928 15:44:57.796022 (+ 4439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:57.796032 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:57.796143 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:57.796256 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:57.799085 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.801427 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.806652 (+ 5225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.807404 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.807546 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.815544 (+ 7998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:57.815562 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:57.815569 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:57.815638 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:57.821251 (+ 5613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.822673 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.822680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.822684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.822912 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.823797 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.823802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.823806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.824230 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:57.824425 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:57.828215 (+ 3790us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.828893 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5605,"thread_start_us":5459,"threads_started":1}
Sep 28 15:44:57 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610297, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.125@KRBTEST.COM
I20260928 15:44:57.830999 20594 heartbeater.cc:347] Connected to a master server at 127.13.5.124:33123
I20260928 15:44:57.831398 20594 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:57.832450 20594 heartbeater.cc:511] Master 127.13.5.124:33123 requested a full tablet report, sending...
I20260928 15:44:57.834420 20270 ts_manager.cc:194] Registered new tserver with Master: dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093)
I20260928 15:44:57.835557 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.794602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.66:50389)
0928 15:44:57.806690 (+ 12088us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:57.806711 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:57.806931 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:57.808617 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:57.808622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:57.809852 (+ 1230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:57.810956 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.810991 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.815444 (+ 4453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.815449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.817735 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.817753 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.819692 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.819701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.820107 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.821099 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:57.821133 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:57.825406 (+ 4273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:57.832460 (+ 7054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.832468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.832474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.833136 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.833622 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.833627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.833631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.833795 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.834582 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:57.834586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:57.834737 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 15:44:57.834868 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:44:57.835174 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11891,"thread_start_us":11596,"threads_started":1}
I20260928 15:44:57.836339 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.788981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.13.5.67:43295)
0928 15:44:57.799360 (+ 10379us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:57.799374 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:57.800957 (+ 1583us) server_negotiation.cc:407] Connection header received
0928 15:44:57.801319 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:57.801334 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:57.801694 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:57.801914 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:57.806010 (+ 4096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.808312 (+ 2302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.810784 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.811950 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.812118 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.821292 (+ 9174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:57.821313 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:57.821335 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:57.821427 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:57.830941 (+ 9514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.832068 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.832077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.832080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.832267 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.832751 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.832758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.832764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.833239 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:57.833636 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:57.835271 (+ 1635us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.835626 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10098,"thread_start_us":9795,"threads_started":1}
I20260928 15:44:57.836627 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.792210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:50389 (local address 127.13.5.126:36553)
0928 15:44:57.800368 (+ 8158us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:57.800377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:57.806797 (+ 6420us) server_negotiation.cc:407] Connection header received
0928 15:44:57.807098 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:57.807104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:57.807196 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:57.807278 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:57.811241 (+ 3963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.812945 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.817927 (+ 4982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.818955 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.819129 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.825792 (+ 6663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:57.825801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:57.825806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:57.825856 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:57.831150 (+ 5294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.833302 (+ 2152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.833308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.833311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.833477 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.833930 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.833934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.833937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.834312 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:57.834461 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:57.835963 (+ 1502us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.836195 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7994,"thread_start_us":7877,"threads_started":1}
I20260928 15:44:57.837767 20595 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36553
I20260928 15:44:57.837926 20595 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:57.838155 19932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:57.838425 20595 heartbeater.cc:511] Master 127.13.5.126:36553 requested a full tablet report, sending...
I20260928 15:44:57.839581 19479 ts_manager.cc:194] Re-registered known tserver with Master: dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093)
I20260928 15:44:57.840560 19479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:50389
I20260928 15:44:57.840662 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.788013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.13.5.66:51093)
0928 15:44:57.794426 (+ 6413us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:57.794453 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:57.795118 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:57.807579 (+ 12461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:57.807585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:57.809110 (+ 1525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:57.810012 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.810028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.819501 (+ 9473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.819506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.822745 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:57.822772 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.825569 (+ 2797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.825581 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:57.825866 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.826919 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:57.826946 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:57.831759 (+ 4813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:57.836336 (+ 4577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.836342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.836346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.836962 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.837759 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:57.837764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:57.837767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:57.837967 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:57.839168 (+ 1201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:57.839176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:57.839371 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 15:44:57.839554 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 15:44:57.840030 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6274,"thread_start_us":6102,"threads_started":1}
I20260928 15:44:57.842357 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:57.785325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51093 (local address 127.13.5.125:45387)
0928 15:44:57.807182 (+ 21857us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:57.807188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:57.807209 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:44:57.807269 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:57.807274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:57.807425 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:57.807563 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:57.813299 (+ 5736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.819278 (+ 5979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.822962 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:57.825143 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:57.826346 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:57.832063 (+ 5717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:57.832069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:57.832073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:57.832119 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:57.836143 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.837397 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.837406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.837411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.837579 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:57.838134 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:57.838141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:57.838144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:57.838560 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:57.838767 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:57.841004 (+ 2237us) server_negotiation.cc:299] Negotiation successful
0928 15:44:57.841323 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21682,"thread_start_us":22409,"threads_started":1}
I20260928 15:44:57.844358 20596 heartbeater.cc:347] Connected to a master server at 127.13.5.125:45387
I20260928 15:44:57.844512 20596 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:57.845050 20596 heartbeater.cc:511] Master 127.13.5.125:45387 requested a full tablet report, sending...
I20260928 15:44:57.846261 20080 ts_manager.cc:194] Registered new tserver with Master: dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093)
I20260928 15:44:57.942057 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 19809
I20260928 15:44:57.959307 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:43295
--local_ip_for_outbound_sockets=127.13.5.67
--tserver_master_addrs=127.13.5.126:36553
--webserver_port=44669
--webserver_interface=127.13.5.67
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab
--principal=oryx/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387,127.13.5.124:33123 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:44:58.217818 20450 heartbeater.cc:503] Master 127.13.5.126:36553 was elected leader, sending a full tablet report...
W20260928 15:44:58.304077 20607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:44:58.304373 20607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:44:58.304471 20607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:44:58.312674 20607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:44:58.312798 20607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:44:58.312915 20607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:44:58.323822 20607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab
--principal=oryx/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:43295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=44669
--tserver_master_addrs=127.13.5.126:36553,127.13.5.125:45387,127.13.5.124:33123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:44:58.325361 20607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:44:58.326984 20607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:44:58.342057 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:58.342929 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:44:58.342945 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:44:58.343320 20607 server_base.cc:1034] running on GCE node
I20260928 15:44:58.344074 20607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:44:58.345142 20607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:44:58.346354 20607 hybrid_clock.cc:648] HybridClock initialized: now 1790610298346292 us; error 55 us; skew 500 ppm
I20260928 15:44:58.346771 20607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:44:58 dist-test-slave-j968 krb5kdc[19435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610298, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:44:58.356305 20607 init.cc:377] Logged in from keytab as oryx/127.13.5.67@KRBTEST.COM (short username oryx)
I20260928 15:44:58.358000 20607 webserver.cc:460] Webserver started at http://127.13.5.67:44669/ using document root <none> and password file <none>
I20260928 15:44:58.358796 20607 fs_manager.cc:359] Metadata directory not provided
I20260928 15:44:58.358960 20607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:44:58.365237 20607 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260928 15:44:58.368526 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:58.370049 20607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:44:58.370239 20607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "c830cd92804c4c96bf17a480cf896d6d"
format_stamp: "Formatted at 2026-09-28 15:44:54 on dist-test-slave-j968"
server_key: "b3de6fae21c984f1cbd99da75dcdd654"
server_key_iv: "1b3581c6be8d88cca85fd665e87b34f0"
server_key_version: "encryptionkey@0"
I20260928 15:44:58.371253 20607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:44:58.390694 20607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:44:58.395239 20607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:44:58.395555 20607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:44:58.396790 20607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:44:58.398264 20607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:44:58.398391 20607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:58.398624 20607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:44:58.398718 20607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:44:58.433398 20607 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:43295
I20260928 15:44:58.433444 20734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:43295 every 8 connection(s)
I20260928 15:44:58.434873 20607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:44:58.441939 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 20607
Sep 28 15:44:58 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610298, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.124@KRBTEST.COM
Sep 28 15:44:58 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610298, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
Sep 28 15:44:58 dist-test-slave-j968 krb5kdc[19435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610298, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.125@KRBTEST.COM
I20260928 15:44:58.470986 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.438637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33123 (local address 127.13.5.67:39803)
0928 15:44:58.439699 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.439770 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.442897 (+ 3127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.445075 (+ 2178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.445092 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.446092 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:58.447141 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.447173 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.449241 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.449250 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.451908 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.451923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.453207 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.453215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.453577 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.454439 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:58.454471 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:58.458449 (+ 3978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:58.464481 (+ 6032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:58.464496 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:58.464519 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:58.465209 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:58.465689 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:58.465696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:58.465703 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:58.466003 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:58.466763 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:58.466772 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:58.467282 (+ 510us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.467631 (+ 349us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.469353 (+ 1722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":939,"threads_started":1}
I20260928 15:44:58.472716 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.439345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:39803 (local address 127.13.5.124:33123)
0928 15:44:58.444499 (+ 5154us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.444504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.444521 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:44:58.444574 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.444577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.444638 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.444771 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:58.447424 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.448995 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.452110 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.452796 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.452911 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.458799 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:58.458808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:58.458813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:58.458892 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:58.464294 (+ 5402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.465388 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.465396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.465403 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.465559 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.466167 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.466173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.466176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.466602 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:58.471858 (+ 5256us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.472051 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.472305 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5037,"thread_start_us":4947,"threads_started":1}
I20260928 15:44:58.474466 20735 heartbeater.cc:347] Connected to a master server at 127.13.5.124:33123
I20260928 15:44:58.474954 20735 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:58.476248 20735 heartbeater.cc:511] Master 127.13.5.124:33123 requested a full tablet report, sending...
I20260928 15:44:58.478631 20270 ts_manager.cc:194] Registered new tserver with Master: c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295)
I20260928 15:44:58.484622 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.441030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.67:41937)
0928 15:44:58.452556 (+ 11526us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.452573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.452752 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.453332 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.453337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.454584 (+ 1247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:58.455027 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.455041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.456520 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.456524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.458116 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.458133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.459616 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.459623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.459821 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.460496 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:58.460524 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:58.467052 (+ 6528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:58.481296 (+ 14244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:58.481305 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:58.481310 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:58.482403 (+ 1093us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:58.482784 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:58.482791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:58.482796 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:58.482952 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:58.483538 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:58.483542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:58.483705 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.483878 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.484174 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11400,"thread_start_us":10963,"threads_started":1}
I20260928 15:44:58.484614 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.441102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:41937 (local address 127.13.5.126:36553)
0928 15:44:58.444136 (+ 3034us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.444146 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.452657 (+ 8511us) server_negotiation.cc:407] Connection header received
0928 15:44:58.452905 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.452911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.453022 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.453107 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:58.455210 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.456328 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.458296 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.458910 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.459143 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.467327 (+ 8184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:58.467334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:58.467340 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:58.467384 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:58.481092 (+ 13708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.482522 (+ 1430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.482528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.482531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.482685 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.483057 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.483063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.483066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.483432 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:58.483569 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.484014 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.484243 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2871,"thread_start_us":2757,"threads_started":1}
I20260928 15:44:58.485528 20736 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36553
I20260928 15:44:58.485684 20736 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:58.486176 20736 heartbeater.cc:511] Master 127.13.5.126:36553 requested a full tablet report, sending...
I20260928 15:44:58.486665 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.439441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.13.5.67:58615)
0928 15:44:58.451772 (+ 12331us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.451793 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.451938 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.453666 (+ 1728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.453674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.454398 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:44:58.454874 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.454897 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.458720 (+ 3823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.458728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.460774 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.460800 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.462212 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.462219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.462450 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.463041 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:44:58.463073 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:44:58.466668 (+ 3595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:44:58.481345 (+ 14677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:58.481375 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:58.481384 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:58.482050 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:58.483253 (+ 1203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:44:58.483258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:44:58.483262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:44:58.483416 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:44:58.485635 (+ 2219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:44:58.485639 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:44:58.485798 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.485948 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.486268 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12181,"thread_start_us":12837,"threads_started":1}
I20260928 15:44:58.487196 19479 ts_manager.cc:194] Re-registered known tserver with Master: c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295)
I20260928 15:44:58.487639 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.437754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:58615 (local address 127.13.5.125:45387)
0928 15:44:58.453079 (+ 15325us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.453088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.453116 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:44:58.453210 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.453218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.453329 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.454226 (+ 897us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:58.457350 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.458530 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.461098 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.461995 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.462753 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.467491 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:58.467500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:58.467505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:58.467549 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:58.481210 (+ 13661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.482610 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.482616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.482619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.483125 (+ 506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.485027 (+ 1902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.485031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.485035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.485489 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:58.486898 (+ 1409us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.487029 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.487255 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15133,"thread_start_us":15697,"threads_started":1}
I20260928 15:44:58.488489 19479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:41937
I20260928 15:44:58.488539 20737 heartbeater.cc:347] Connected to a master server at 127.13.5.125:45387
I20260928 15:44:58.488643 20737 heartbeater.cc:464] Registering TS with master...
I20260928 15:44:58.489202 20737 heartbeater.cc:511] Master 127.13.5.125:45387 requested a full tablet report, sending...
I20260928 15:44:58.490396 20080 ts_manager.cc:194] Registered new tserver with Master: c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295)
I20260928 15:44:58.501789 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.468590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.13.5.126:36553)
0928 15:44:58.480613 (+ 12023us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.480620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.480641 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:44:58.480705 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.480710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.480793 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.480907 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:58.483671 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.485082 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.487315 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.491037 (+ 3722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.491158 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.494752 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:58.494760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:58.494765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:58.494806 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:58.498486 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.499291 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.499299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.499303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.499467 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.499975 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.499981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.499986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.500355 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:58.500690 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.501124 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.501362 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2869,"server-negotiator.queue_time_us":11900,"thread_start_us":11766,"threads_started":1}
I20260928 15:44:58.503940 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.468105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.13.5.125:45387)
0928 15:44:58.482413 (+ 14308us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.482422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.482443 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:44:58.482502 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.482508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.482603 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.483675 (+ 1072us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:58.486293 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.488440 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.490726 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.491399 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.491551 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.495467 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:58.495475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:58.495479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:58.495522 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:58.500638 (+ 5116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.501578 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.501584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.501588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.501731 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.502212 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.502219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.502223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.502560 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:58.502824 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.503254 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.503482 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14171,"thread_start_us":14606,"threads_started":1}
I20260928 15:44:58.504998 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.473188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.13.5.124:33123)
0928 15:44:58.476623 (+ 3435us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.476632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.480680 (+ 4048us) server_negotiation.cc:407] Connection header received
0928 15:44:58.480742 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.480746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.480854 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.481919 (+ 1065us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:58.488932 (+ 7013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.490061 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.492705 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.493485 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.493627 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.497063 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:58.497070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:58.497076 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:58.497121 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:58.500352 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.501391 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.501399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.501403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.501711 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:58.502167 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:58.502173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:58.502178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:58.502553 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:58.503016 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.504589 (+ 1573us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.504763 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3031}
I20260928 15:44:58.506075 19479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:44:58.513484 19479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:44:58.554540 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.535116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.13.5.67:43295)
0928 15:44:58.544412 (+ 9296us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.544422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.544457 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:44:58.544547 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.544554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.544800 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.545011 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:58.545861 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.547951 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.551332 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.553309 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.553462 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.553714 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.553997 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.554197 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9106,"thread_start_us":8929,"threads_started":1}
I20260928 15:44:58.554605 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.534700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:43295 (local address 127.0.0.1:59528)
0928 15:44:58.542776 (+ 8076us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.542804 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.542964 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.545032 (+ 2068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.545041 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.545080 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:58.545674 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.545688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.548109 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.548115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.551088 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.551104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.553532 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.553541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.553768 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.553819 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.553921 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.554098 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7902,"mutex_wait_us":56,"thread_start_us":8595,"threads_started":1}
I20260928 15:44:58.558141 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.534248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:33255 (local address 127.0.0.1:45504)
0928 15:44:58.540147 (+ 5899us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.540175 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.540341 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.546339 (+ 5998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.546347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.546376 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:58.547175 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.547189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.550498 (+ 3309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.550508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.553321 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.553334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.557097 (+ 3763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.557106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.557334 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.557385 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.557483 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.557658 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":468,"threads_started":1}
I20260928 15:44:58.558243 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.534507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.13.5.65:33255)
0928 15:44:58.545778 (+ 11271us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.545788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.545816 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:44:58.545906 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.545915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.546137 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.546502 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:58.548820 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.550378 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.554830 (+ 4452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.556563 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.556698 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.556944 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.557551 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.557733 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11074,"thread_start_us":10986,"threads_started":1}
I20260928 15:44:58.559695 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.536223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.13.5.66:43093)
0928 15:44:58.536759 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.536771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.543705 (+ 6934us) server_negotiation.cc:407] Connection header received
0928 15:44:58.543929 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.543941 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.544275 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.544502 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:58.546876 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.548619 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.555567 (+ 6948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.557020 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.557127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.557346 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.559153 (+ 1807us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.559339 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":254,"threads_started":1}
I20260928 15:44:58.559779 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.536031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43093 (local address 127.0.0.1:33308)
0928 15:44:58.543583 (+ 7552us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.543597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.543745 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.544729 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.544734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.544758 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:58.545193 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.545207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.548790 (+ 3583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.548795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.554677 (+ 5882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.554697 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.558650 (+ 3953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.558659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.558960 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.559009 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.559101 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.559230 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7446,"thread_start_us":6344,"threads_started":1}
I20260928 15:44:58.561028 20670 tablet_service.cc:1467] Processing CreateTablet for tablet dde0c88491564b758695e14cc31da959 (DEFAULT_TABLE table=test-table [id=a778ac4497b7407bb15ad4f2e3503136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:58.562065 20384 tablet_service.cc:1467] Processing CreateTablet for tablet dde0c88491564b758695e14cc31da959 (DEFAULT_TABLE table=test-table [id=a778ac4497b7407bb15ad4f2e3503136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:58.562824 20670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde0c88491564b758695e14cc31da959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:58.563557 20384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde0c88491564b758695e14cc31da959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:58.565356 20529 tablet_service.cc:1467] Processing CreateTablet for tablet dde0c88491564b758695e14cc31da959 (DEFAULT_TABLE table=test-table [id=a778ac4497b7407bb15ad4f2e3503136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:44:58.566601 20529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde0c88491564b758695e14cc31da959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:44:58.587586 20772 tablet_bootstrap.cc:492] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Bootstrap starting.
I20260928 15:44:58.592180 20771 tablet_bootstrap.cc:492] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Bootstrap starting.
I20260928 15:44:58.593591 20772 tablet_bootstrap.cc:654] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:58.593612 20773 tablet_bootstrap.cc:492] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Bootstrap starting.
I20260928 15:44:58.595299 20772 log.cc:824] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:58.598286 20771 tablet_bootstrap.cc:654] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:58.598842 20773 tablet_bootstrap.cc:654] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Neither blocks nor log segments found. Creating new log.
I20260928 15:44:58.599179 20772 tablet_bootstrap.cc:492] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: No bootstrap required, opened a new log
I20260928 15:44:58.599485 20772 ts_tablet_manager.cc:1397] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 15:44:58.600279 20771 log.cc:824] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:58.600896 20773 log.cc:824] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Log is configured to *not* fsync() on all Append() calls
I20260928 15:44:58.604128 20771 tablet_bootstrap.cc:492] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: No bootstrap required, opened a new log
I20260928 15:44:58.604496 20771 ts_tablet_manager.cc:1397] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.008s
I20260928 15:44:58.604570 20773 tablet_bootstrap.cc:492] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: No bootstrap required, opened a new log
I20260928 15:44:58.604928 20773 ts_tablet_manager.cc:1397] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260928 15:44:58.612638 20772 raft_consensus.cc:348] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.613256 20772 raft_consensus.cc:374] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:58.613430 20772 raft_consensus.cc:729] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c830cd92804c4c96bf17a480cf896d6d, State: Initialized, Role: FOLLOWER
I20260928 15:44:58.614326 20772 consensus_queue.cc:260] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.616438 20771 raft_consensus.cc:348] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.616932 20772 ts_tablet_manager.cc:1428] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260928 15:44:58.616986 20771 raft_consensus.cc:374] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:58.617156 20771 raft_consensus.cc:729] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b75b8b58bd4425499d56d72b7158364, State: Initialized, Role: FOLLOWER
I20260928 15:44:58.617280 20736 heartbeater.cc:503] Master 127.13.5.126:36553 was elected leader, sending a full tablet report...
I20260928 15:44:58.617844 20771 consensus_queue.cc:260] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.618824 20773 raft_consensus.cc:348] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.619479 20773 raft_consensus.cc:374] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:44:58.619613 20773 raft_consensus.cc:729] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbdcc4f879041f8953c4b203a208781, State: Initialized, Role: FOLLOWER
I20260928 15:44:58.620510 20773 consensus_queue.cc:260] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.623175 20595 heartbeater.cc:503] Master 127.13.5.126:36553 was elected leader, sending a full tablet report...
I20260928 15:44:58.629557 20771 ts_tablet_manager.cc:1428] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Time spent starting tablet: real 0.025s user 0.012s sys 0.012s
I20260928 15:44:58.628546 20773 ts_tablet_manager.cc:1428] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Time spent starting tablet: real 0.024s user 0.011s sys 0.008s
W20260928 15:44:58.650391 20452 tablet.cc:2369] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:58.688690 20738 tablet.cc:2369] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:44:58.791150 20597 tablet.cc:2369] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:44:58.941293 20779 raft_consensus.cc:482] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:58.941764 20779 raft_consensus.cc:504] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.942596 20777 raft_consensus.cc:482] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:44:58.942972 20777 raft_consensus.cc:504] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.944306 20779 leader_election.cc:288] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255), c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295)
I20260928 15:44:58.944991 20777 leader_election.cc:288] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255), dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093)
I20260928 15:44:58.960116 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.945398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:33255 (local address 127.13.5.67:47091)
0928 15:44:58.946170 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.946193 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.946343 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.949191 (+ 2848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.949198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.949231 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:58.949468 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.949481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.954000 (+ 4519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.954005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.956433 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.956450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.959114 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.959121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.959341 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.959391 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.959492 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.959649 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":4}
I20260928 15:44:58.960526 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.946062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:47091 (local address 127.13.5.65:33255)
0928 15:44:58.947157 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.947166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.947191 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:44:58.947286 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.947294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.947386 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.947491 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:58.951602 (+ 4111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.953294 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.956724 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.958294 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.958426 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.958552 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.959564 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.959777 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876}
I20260928 15:44:58.962281 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde0c88491564b758695e14cc31da959" candidate_uuid: "c830cd92804c4c96bf17a480cf896d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b75b8b58bd4425499d56d72b7158364" is_pre_election: true
I20260928 15:44:58.963335 20404 raft_consensus.cc:2454] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c830cd92804c4c96bf17a480cf896d6d in term 0.
I20260928 15:44:58.963577 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.945678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43093 (local address 127.13.5.67:51005)
0928 15:44:58.946708 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.946727 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.946823 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.948598 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.948606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.948644 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:58.948995 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.949011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.954873 (+ 5862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.954877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.959832 (+ 4955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.959881 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.962543 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.962553 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.962889 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.962930 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.963005 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.963157 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":2783}
I20260928 15:44:58.964537 20624 leader_election.cc:302] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b75b8b58bd4425499d56d72b7158364, c830cd92804c4c96bf17a480cf896d6d; no voters:
I20260928 15:44:58.964781 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.947783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:51005 (local address 127.13.5.66:43093)
0928 15:44:58.948047 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.948056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.948080 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:44:58.948166 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.948174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.948270 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.948375 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:58.949663 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.951276 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.960133 (+ 8857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.962264 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.963988 (+ 1724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.964095 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.964150 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.964316 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 15:44:58.965234 20777 raft_consensus.cc:2781] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:44:58.965421 20777 raft_consensus.cc:482] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:44:58.965544 20777 raft_consensus.cc:3037] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:58.965818 20549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde0c88491564b758695e14cc31da959" candidate_uuid: "c830cd92804c4c96bf17a480cf896d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbdcc4f879041f8953c4b203a208781" is_pre_election: true
I20260928 15:44:58.966415 20549 raft_consensus.cc:2454] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c830cd92804c4c96bf17a480cf896d6d in term 0.
I20260928 15:44:58.967617 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.945346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:58209 (local address 127.13.5.67:43295)
0928 15:44:58.946022 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.946031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.954504 (+ 8473us) server_negotiation.cc:407] Connection header received
0928 15:44:58.954577 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.954581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.954650 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.954731 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:58.956464 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.957723 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.964373 (+ 6650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.965902 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.966002 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.966098 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.967149 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.967332 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
I20260928 15:44:58.968497 20780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.944582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:43295 (local address 127.13.5.66:58209)
0928 15:44:58.954001 (+ 9419us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.954021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.954197 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.955825 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.955830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.955859 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:58.956274 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.956288 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.957937 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.957943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.964173 (+ 6230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.964190 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.966684 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.966691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.966945 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.966979 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.967809 (+ 830us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.968071 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9185,"thread_start_us":9702,"threads_started":1}
I20260928 15:44:58.969439 20690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde0c88491564b758695e14cc31da959" candidate_uuid: "dfbdcc4f879041f8953c4b203a208781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c830cd92804c4c96bf17a480cf896d6d" is_pre_election: true
I20260928 15:44:58.969802 20777 raft_consensus.cc:504] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.970165 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.946230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:60457 (local address 127.13.5.65:33255)
0928 15:44:58.958965 (+ 12735us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:58.958971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:58.961039 (+ 2068us) server_negotiation.cc:407] Connection header received
0928 15:44:58.961264 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:58.961269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:58.961363 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:58.961493 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:44:58.962711 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.964271 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.967507 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.968928 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.969083 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.969283 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:58.969644 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 15:44:58.969780 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12557,"thread_start_us":12753,"threads_started":1}
I20260928 15:44:58.970203 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:58.944674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:33255 (local address 127.13.5.66:60457)
0928 15:44:58.960950 (+ 16276us) negotiation.cc:107] Waiting for socket to connect
0928 15:44:58.960970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:44:58.961144 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:44:58.961552 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:44:58.961562 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:44:58.961593 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:44:58.962219 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.962236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.964420 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.964427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.967312 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:44:58.967326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:44:58.969199 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:44:58.969207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:44:58.969475 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:58.969500 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 15:44:58.969616 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:44:58.969756 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16080,"thread_start_us":15772,"threads_started":1}
I20260928 15:44:58.970404 20690 raft_consensus.cc:2379] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dfbdcc4f879041f8953c4b203a208781 in current term 1: Already voted for candidate c830cd92804c4c96bf17a480cf896d6d in this term.
I20260928 15:44:58.971175 20777 leader_election.cc:288] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [CANDIDATE]: Term 1 election: Requested vote from peers 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255), dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093)
I20260928 15:44:58.971180 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde0c88491564b758695e14cc31da959" candidate_uuid: "c830cd92804c4c96bf17a480cf896d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b75b8b58bd4425499d56d72b7158364"
I20260928 15:44:58.971349 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde0c88491564b758695e14cc31da959" candidate_uuid: "dfbdcc4f879041f8953c4b203a208781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b75b8b58bd4425499d56d72b7158364" is_pre_election: true
I20260928 15:44:58.971491 20404 raft_consensus.cc:3037] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:58.971747 20549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde0c88491564b758695e14cc31da959" candidate_uuid: "c830cd92804c4c96bf17a480cf896d6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbdcc4f879041f8953c4b203a208781"
I20260928 15:44:58.972050 20549 raft_consensus.cc:3037] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:44:58.972766 20483 leader_election.cc:302] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfbdcc4f879041f8953c4b203a208781; no voters: 5b75b8b58bd4425499d56d72b7158364, c830cd92804c4c96bf17a480cf896d6d
I20260928 15:44:58.975036 20404 raft_consensus.cc:2454] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c830cd92804c4c96bf17a480cf896d6d in term 1.
I20260928 15:44:58.975462 20549 raft_consensus.cc:2454] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c830cd92804c4c96bf17a480cf896d6d in term 1.
I20260928 15:44:58.975476 20624 leader_election.cc:302] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b75b8b58bd4425499d56d72b7158364, c830cd92804c4c96bf17a480cf896d6d; no voters:
I20260928 15:44:58.975939 20779 raft_consensus.cc:2726] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:44:58.976828 20777 raft_consensus.cc:2781] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:44:58.977326 20777 raft_consensus.cc:686] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 1 LEADER]: Becoming Leader. State: Replica: c830cd92804c4c96bf17a480cf896d6d, State: Running, Role: LEADER
I20260928 15:44:58.978008 20777 consensus_queue.cc:237] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:44:58.984037 19478 catalog_manager.cc:5495] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d reported cstate change: term changed from 0 to 1, leader changed from <none> to c830cd92804c4c96bf17a480cf896d6d (127.13.5.67). New cstate: current_term: 1 leader_uuid: "c830cd92804c4c96bf17a480cf896d6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } health_report { overall_health: UNKNOWN } } }
I20260928 15:44:59.019436 20768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:59.012722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.13.5.67:43295)
0928 15:44:59.012926 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:59.012933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:59.013379 (+ 446us) server_negotiation.cc:407] Connection header received
0928 15:44:59.013566 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:59.013570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:59.013650 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:59.013777 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:44:59.014317 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.015294 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.016952 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.017531 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.017649 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:59.018203 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:44:59.018767 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:44:59.018899 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:59.018966 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 15:44:59.019060 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:44:59.029356 20404 raft_consensus.cc:1261] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 1 FOLLOWER]: Refusing update from remote peer c830cd92804c4c96bf17a480cf896d6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:59.029402 20549 raft_consensus.cc:1261] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Refusing update from remote peer c830cd92804c4c96bf17a480cf896d6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:44:59.030537 20777 consensus_queue.cc:1035] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:59.030869 20783 consensus_queue.cc:1035] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:44:59.049007 20787 mvcc.cc:205] Tried to move back new op lower bound from 7334339784812625920 to 7334339784618115072. Current Snapshot: MvccSnapshot[applied={T|T < 7334339784812625920}]
I20260928 15:44:59.052227 20788 mvcc.cc:205] Tried to move back new op lower bound from 7334339784812625920 to 7334339784618115072. Current Snapshot: MvccSnapshot[applied={T|T < 7334339784812625920}]
I20260928 15:44:59.070820 19478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:59.071164 19478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:44:59.079360 19478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2722471774848d0923a854406fc2ba2: Sending DeleteTablet for 3 replicas of tablet dde0c88491564b758695e14cc31da959
I20260928 15:44:59.080678 20384 tablet_service.cc:1514] Processing DeleteTablet for tablet dde0c88491564b758695e14cc31da959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:59 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:45504
I20260928 15:44:59.080761 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet dde0c88491564b758695e14cc31da959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:59 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:59528
I20260928 15:44:59.081916 20528 tablet_service.cc:1514] Processing DeleteTablet for tablet dde0c88491564b758695e14cc31da959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:59 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:33308
I20260928 15:44:59.082386 20529 tablet_service.cc:1514] Processing DeleteTablet for tablet dde0c88491564b758695e14cc31da959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:44:59 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:33308
I20260928 15:44:59.083585 20798 tablet_replica.cc:330] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: stopping tablet replica
I20260928 15:44:59.084303 20798 raft_consensus.cc:2229] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:44:59.085284 20798 raft_consensus.cc:2258] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:59.088374 20798 ts_tablet_manager.cc:1905] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:59.092679 20797 tablet_replica.cc:330] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: stopping tablet replica
I20260928 15:44:59.093281 20800 tablet_replica.cc:330] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: stopping tablet replica
I20260928 15:44:59.093266 20797 raft_consensus.cc:2229] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:59.093791 20800 raft_consensus.cc:2229] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:44:59.094038 20797 raft_consensus.cc:2258] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:59.094722 20800 raft_consensus.cc:2258] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:44:59.097117 20800 ts_tablet_manager.cc:1905] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:59.100098 20797 ts_tablet_manager.cc:1905] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:44:59.106061 20800 ts_tablet_manager.cc:1918] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:59.106261 20800 log.cc:1196] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/dde0c88491564b758695e14cc31da959
I20260928 15:44:59.107023 20800 ts_tablet_manager.cc:1939] T dde0c88491564b758695e14cc31da959 P dfbdcc4f879041f8953c4b203a208781: Deleting consensus metadata
I20260928 15:44:59.109473 19466 catalog_manager.cc:4841] TS dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093): tablet dde0c88491564b758695e14cc31da959 (table test-table [id=a778ac4497b7407bb15ad4f2e3503136]) successfully deleted
W20260928 15:44:59.110626 19466 catalog_manager.cc:4808] TS dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093): delete failed for tablet dde0c88491564b758695e14cc31da959 because the tablet was not found. No further retry: Not found: Tablet not found: dde0c88491564b758695e14cc31da959
I20260928 15:44:59.112221 20798 ts_tablet_manager.cc:1918] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:59.112552 20798 log.cc:1196] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/dde0c88491564b758695e14cc31da959
I20260928 15:44:59.113354 20798 ts_tablet_manager.cc:1939] T dde0c88491564b758695e14cc31da959 P c830cd92804c4c96bf17a480cf896d6d: Deleting consensus metadata
I20260928 15:44:59.115329 19467 catalog_manager.cc:4841] TS c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295): tablet dde0c88491564b758695e14cc31da959 (table test-table [id=a778ac4497b7407bb15ad4f2e3503136]) successfully deleted
I20260928 15:44:59.115929 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 19448
I20260928 15:44:59.117013 20797 ts_tablet_manager.cc:1918] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:44:59.117061 19510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:44:59.117168 20797 log.cc:1196] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/dde0c88491564b758695e14cc31da959
I20260928 15:44:59.118439 20797 ts_tablet_manager.cc:1939] T dde0c88491564b758695e14cc31da959 P 5b75b8b58bd4425499d56d72b7158364: Deleting consensus metadata
I20260928 15:44:59.134384 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:59.082931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.13.5.125:45387)
0928 15:44:59.098040 (+ 15109us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:59.098050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:59.107456 (+ 9406us) server_negotiation.cc:407] Connection header received
0928 15:44:59.107786 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:59.107792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:59.107898 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:59.108016 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:59.111655 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.113691 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.119275 (+ 5584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.120272 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.120457 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:59.125125 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:59.125132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:59.125137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:59.125184 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:59.129723 (+ 4539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:59.130843 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:59.130850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:59.130853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:59.131021 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:59.131494 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:59.131500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:59.131504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:59.131934 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:59.132196 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:59.133700 (+ 1504us) server_negotiation.cc:299] Negotiation successful
0928 15:44:59.133982 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14754,"thread_start_us":15036,"threads_started":1}
I20260928 15:44:59.154312 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:59.107443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.13.5.124:33123)
0928 15:44:59.122739 (+ 15296us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:59.122746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:59.133368 (+ 10622us) server_negotiation.cc:407] Connection header received
0928 15:44:59.133566 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:59.133573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:59.133687 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:59.133803 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:59.135851 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.137359 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.139962 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.140678 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.140805 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:59.144747 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:59.144756 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:59.144762 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:59.144818 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:59.149707 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:59.150654 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:59.150660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:59.150664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:59.150836 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:59.151307 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:59.151312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:59.151316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:59.151669 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:59.152045 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:59.153438 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0928 15:44:59.153738 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15110,"thread_start_us":15302,"threads_started":1}
I20260928 15:44:59.271108 19465 catalog_manager.cc:4841] TS 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255): tablet dde0c88491564b758695e14cc31da959 (table test-table [id=a778ac4497b7407bb15ad4f2e3503136]) successfully deleted
I20260928 15:44:59.271809 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 19448
I20260928 15:44:59.313601 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:59.296242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.13.5.125:45387)
0928 15:44:59.296433 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:59.296440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:59.297665 (+ 1225us) server_negotiation.cc:407] Connection header received
0928 15:44:59.298017 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:59.298022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:59.298131 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:59.298232 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:59.300415 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.301166 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.303049 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.303485 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.303624 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:59.307088 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:59.307098 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:59.307102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:59.307138 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:59.310404 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:59.311218 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:59.311224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:59.311228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:59.311451 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:59.311904 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:59.311909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:59.311912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:59.312283 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:59.312582 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:59.312988 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 15:44:59.313224 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:44:59.316123 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:44:59.298030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.13.5.124:33123)
0928 15:44:59.298248 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:44:59.298254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:44:59.298318 (+ 64us) server_negotiation.cc:407] Connection header received
0928 15:44:59.298588 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:44:59.298596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:44:59.298688 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:44:59.298811 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:44:59.301496 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.302619 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.304759 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:44:59.305501 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:44:59.305658 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:44:59.309133 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:44:59.309139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:44:59.309143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:44:59.309189 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:44:59.312778 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:59.313664 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:59.313670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:59.313674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:59.313872 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:44:59.314298 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:44:59.314303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:44:59.314307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:44:59.314650 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:44:59.314894 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 15:44:59.315522 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 15:44:59.315708 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:45:00.065344 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.064504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.67:45823)
0928 15:45:00.065046 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.065177 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":302,"threads_started":1}
W20260928 15:45:00.065775 20736 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:36553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
I20260928 15:45:00.067390 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.066572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.65:52705)
0928 15:45:00.067123 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.067229 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":248,"threads_started":1}
W20260928 15:45:00.067839 20450 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:36553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
I20260928 15:45:00.084738 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.083915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.66:54061)
0928 15:45:00.084441 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.084560 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
W20260928 15:45:00.085273 20595 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:36553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
I20260928 15:45:00.578775 20813 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2722471774848d0923a854406fc2ba2)
I20260928 15:45:00.579057 20813 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } }
I20260928 15:45:00.579399 20814 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2722471774848d0923a854406fc2ba2)
I20260928 15:45:00.579700 20814 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } }
I20260928 15:45:00.580180 20813 leader_election.cc:288] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2722471774848d0923a854406fc2ba2 (127.13.5.126:36553), ba69234c12ab4e909a9e6bfc2b2843fe (127.13.5.125:45387)
I20260928 15:45:00.587898 20814 leader_election.cc:288] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2722471774848d0923a854406fc2ba2 (127.13.5.126:36553), fdfb7623849d4bb1859a7f7f15d2ba90 (127.13.5.124:33123)
I20260928 15:45:00.591709 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.580256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58474)
0928 15:45:00.591160 (+ 10904us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.591394 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10733,"thread_start_us":10915,"threads_started":1}
W20260928 15:45:00.592167 20255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
I20260928 15:45:00.593611 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.582044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58486)
0928 15:45:00.591938 (+ 9894us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.593281 (+ 1343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9705,"thread_start_us":9581,"threads_started":1}
W20260928 15:45:00.594089 20066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
I20260928 15:45:00.594466 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.594103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58494)
0928 15:45:00.594234 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.594309 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 15:45:00.594820 20255 leader_election.cc:334] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e2722471774848d0923a854406fc2ba2 (127.13.5.126:36553): Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
I20260928 15:45:00.598292 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.597816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58510)
0928 15:45:00.598001 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.598107 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 15:45:00.598588 20066 leader_election.cc:334] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e2722471774848d0923a854406fc2ba2 (127.13.5.126:36553): Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
I20260928 15:45:00.599144 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.580385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:45387 (local address 127.0.0.1:50032)
0928 15:45:00.591452 (+ 11067us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.591469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:00.591639 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:00.592203 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:00.592208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:00.592241 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:00.592552 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.592562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.594889 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.594898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.596612 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.596625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.598355 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.598361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.598534 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.598563 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:45:00.598622 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 15:45:00.598746 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10932,"mutex_wait_us":3,"thread_start_us":10426,"threads_started":1}
I20260928 15:45:00.599572 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.580567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.13.5.125:45387)
0928 15:45:00.590951 (+ 10384us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.590961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.591881 (+ 920us) server_negotiation.cc:407] Connection header received
0928 15:45:00.591965 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.591969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.592052 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.592183 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:00.592722 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.594685 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.596762 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.598136 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.598899 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.599026 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.599084 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.599242 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10161,"thread_start_us":9932,"threads_started":1}
I20260928 15:45:00.600140 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" is_pre_election: true
I20260928 15:45:00.600446 20090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfb7623849d4bb1859a7f7f15d2ba90 in term 1.
I20260928 15:45:00.601035 20256 leader_election.cc:302] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba69234c12ab4e909a9e6bfc2b2843fe, fdfb7623849d4bb1859a7f7f15d2ba90; no voters: e2722471774848d0923a854406fc2ba2
I20260928 15:45:00.601426 20813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 15:45:00.601572 20813 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2722471774848d0923a854406fc2ba2)
I20260928 15:45:00.601698 20813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:45:00.601727 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.592380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33123 (local address 127.0.0.1:50614)
0928 15:45:00.593195 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.593217 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:00.593339 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:00.594468 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:00.594474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:00.594503 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:00.594960 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.594976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.597307 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.597314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.599187 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.599199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.600850 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.600856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.601039 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.601062 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:45:00.601170 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:45:00.601299 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":1,"thread_start_us":530,"threads_started":1}
I20260928 15:45:00.601840 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.593291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.13.5.124:33123)
0928 15:45:00.594106 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.594114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.594137 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:45:00.594197 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.594201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.594275 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.595160 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:00.595573 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.597167 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.599374 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.600644 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.600813 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.600928 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.601194 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.601320 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":548,"threads_started":1}
I20260928 15:45:00.602406 20280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" is_pre_election: true
I20260928 15:45:00.604077 20813 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } }
I20260928 15:45:00.604590 20280 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ba69234c12ab4e909a9e6bfc2b2843fe in current term 2: Already voted for candidate fdfb7623849d4bb1859a7f7f15d2ba90 in this term.
I20260928 15:45:00.604743 20813 leader_election.cc:288] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [CANDIDATE]: Term 2 election: Requested vote from peers e2722471774848d0923a854406fc2ba2 (127.13.5.126:36553), ba69234c12ab4e909a9e6bfc2b2843fe (127.13.5.125:45387)
I20260928 15:45:00.605352 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.604896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58514)
0928 15:45:00.605080 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.605179 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:45:00.605399 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe"
I20260928 15:45:00.605648 20090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:45:00.605760 20066 leader_election.cc:302] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba69234c12ab4e909a9e6bfc2b2843fe; no voters: e2722471774848d0923a854406fc2ba2, fdfb7623849d4bb1859a7f7f15d2ba90
I20260928 15:45:00.608181 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.607725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58532)
0928 15:45:00.607945 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.608037 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 15:45:00.608455 20255 leader_election.cc:334] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e2722471774848d0923a854406fc2ba2 (127.13.5.126:36553): Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
I20260928 15:45:00.608845 20090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfb7623849d4bb1859a7f7f15d2ba90 in term 2.
I20260928 15:45:00.609026 20814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:45:00.609269 20256 leader_election.cc:302] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ba69234c12ab4e909a9e6bfc2b2843fe, fdfb7623849d4bb1859a7f7f15d2ba90; no voters: e2722471774848d0923a854406fc2ba2
I20260928 15:45:00.609771 20813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:45:00.610426 20813 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [term 2 LEADER]: Becoming Leader. State: Replica: fdfb7623849d4bb1859a7f7f15d2ba90, State: Running, Role: LEADER
I20260928 15:45:00.610891 20813 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } }
I20260928 15:45:00.612867 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdfb7623849d4bb1859a7f7f15d2ba90. Latest consensus state: current_term: 2 leader_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:45:00.613276 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:00.613867 20825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:45:00.617534 20825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:45:00.618340 20825 catalog_manager.cc:1259] Loaded cluster ID: 61c2b26ae621471b90884dc025292186
I20260928 15:45:00.618412 20825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:45:00.619547 20825 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:45:00.620431 20825 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Loaded TSK: 0
I20260928 15:45:00.621295 20825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:45:00.667382 20270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:45:00.674717 20270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:45:00.679397 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.678948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58546)
0928 15:45:00.679142 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.679240 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:45:00.679546 20090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [term 2 FOLLOWER]: Refusing update from remote peer fdfb7623849d4bb1859a7f7f15d2ba90: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 15:45:00.680388 20823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 15:45:00.680658 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.680058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58550)
0928 15:45:00.680339 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.680444 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:45:00.680899 20255 consensus_peers.cc:487] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 -> Peer e2722471774848d0923a854406fc2ba2 (127.13.5.126:36553): Couldn't send request to peer e2722471774848d0923a854406fc2ba2. Status: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 15:45:00.683415 20814 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdfb7623849d4bb1859a7f7f15d2ba90. Latest consensus state: current_term: 2 leader_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:45:00.683682 20814 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:45:00.685449 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:45:00.685829 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90 [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:00.686430 20814 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e2722471774848d0923a854406fc2ba2" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36553 } } peers { permanent_uuid: "ba69234c12ab4e909a9e6bfc2b2843fe" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 45387 } attrs { promote: false } } peers { permanent_uuid: "fdfb7623849d4bb1859a7f7f15d2ba90" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33123 } attrs { promote: false } } }
I20260928 15:45:00.686710 20814 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba69234c12ab4e909a9e6bfc2b2843fe [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:45:00.702456 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.694888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:33255 (local address 127.0.0.1:45506)
0928 15:45:00.695084 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.695113 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:00.695256 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:00.696137 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:00.696142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:00.696172 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:00.696390 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.696402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.697889 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.697895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.699654 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.699666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.701551 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.701559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.701900 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.701939 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:45:00.702030 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:45:00.702171 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 15:45:00.703459 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.694959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.13.5.65:33255)
0928 15:45:00.695617 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.695626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.695648 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:45:00.695740 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.695745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.695827 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.696100 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:00.696560 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.697715 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.699831 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.701210 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.702827 (+ 1617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.702958 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.703011 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.703154 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":274,"threads_started":1}
I20260928 15:45:00.703840 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.695269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:43295 (local address 127.0.0.1:59542)
0928 15:45:00.695603 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.695627 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:00.695736 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:00.696486 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:00.696491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:00.696525 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:00.696837 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.696849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.699293 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.699305 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.701384 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.701399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.703065 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.703072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.703244 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.703279 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:45:00.703392 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 15:45:00.703546 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260928 15:45:00.704316 20384 tablet_service.cc:1467] Processing CreateTablet for tablet e9c8770fd18b4fe1883a0426a228f507 (DEFAULT_TABLE table=test-table [id=084edc2a8dd94618ac9181f59d36c42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:00.704488 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.695613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.13.5.67:43295)
0928 15:45:00.696114 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.696120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.696138 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:45:00.696205 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.696210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.696276 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.696402 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:00.698055 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.699142 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.701550 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.702850 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.702986 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.703085 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.704061 (+ 976us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.704205 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":271,"threads_started":1}
I20260928 15:45:00.704828 20384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c8770fd18b4fe1883a0426a228f507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:00.705210 20670 tablet_service.cc:1467] Processing CreateTablet for tablet e9c8770fd18b4fe1883a0426a228f507 (DEFAULT_TABLE table=test-table [id=084edc2a8dd94618ac9181f59d36c42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:00.705652 20670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c8770fd18b4fe1883a0426a228f507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:00.707404 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.697205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43093 (local address 127.0.0.1:33312)
0928 15:45:00.697864 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.697886 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:00.698033 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:00.700760 (+ 2727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:00.700765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:00.700792 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:00.701264 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.701278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.702740 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.702745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.704739 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.704751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.706636 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.706642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.706816 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.706839 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:45:00.706954 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 15:45:00.707085 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":292,"threads_started":1}
I20260928 15:45:00.707430 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.697358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.13.5.66:43093)
0928 15:45:00.700269 (+ 2911us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.700282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.700301 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:45:00.700380 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.700385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.700497 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.700595 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:00.701507 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.702571 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.704932 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.706445 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.706588 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.706701 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.706982 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.707122 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2779,"thread_start_us":2659,"threads_started":1}
I20260928 15:45:00.708264 20528 tablet_service.cc:1467] Processing CreateTablet for tablet e9c8770fd18b4fe1883a0426a228f507 (DEFAULT_TABLE table=test-table [id=084edc2a8dd94618ac9181f59d36c42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:00.708750 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c8770fd18b4fe1883a0426a228f507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:00.712234 20835 tablet_bootstrap.cc:492] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: Bootstrap starting.
I20260928 15:45:00.713382 20836 tablet_bootstrap.cc:492] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: Bootstrap starting.
I20260928 15:45:00.714623 20835 tablet_bootstrap.cc:654] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:00.714939 20836 tablet_bootstrap.cc:654] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:00.715422 20837 tablet_bootstrap.cc:492] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: Bootstrap starting.
I20260928 15:45:00.716966 20835 tablet_bootstrap.cc:492] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: No bootstrap required, opened a new log
I20260928 15:45:00.717144 20835 ts_tablet_manager.cc:1397] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 15:45:00.717772 20837 tablet_bootstrap.cc:654] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:00.718395 20835 raft_consensus.cc:348] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.718802 20835 raft_consensus.cc:374] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:00.718918 20835 raft_consensus.cc:729] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b75b8b58bd4425499d56d72b7158364, State: Initialized, Role: FOLLOWER
I20260928 15:45:00.719172 20835 consensus_queue.cc:260] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.720294 20835 ts_tablet_manager.cc:1428] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 15:45:00.721437 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.720341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.65:56841)
0928 15:45:00.720990 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.721135 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":350,"threads_started":1}
I20260928 15:45:00.726826 20449 heartbeater.cc:503] Master 127.13.5.124:33123 was elected leader, sending a full tablet report...
I20260928 15:45:00.727329 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.722018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.65:52799)
0928 15:45:00.726703 (+ 4685us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.726907 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4503,"thread_start_us":4354,"threads_started":1}
I20260928 15:45:00.728430 20836 tablet_bootstrap.cc:492] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: No bootstrap required, opened a new log
I20260928 15:45:00.728627 20836 ts_tablet_manager.cc:1397] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20260928 15:45:00.729445 20837 tablet_bootstrap.cc:492] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: No bootstrap required, opened a new log
I20260928 15:45:00.729612 20837 ts_tablet_manager.cc:1397] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20260928 15:45:00.729575 20836 raft_consensus.cc:348] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.729833 20836 raft_consensus.cc:374] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:00.729931 20836 raft_consensus.cc:729] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c830cd92804c4c96bf17a480cf896d6d, State: Initialized, Role: FOLLOWER
I20260928 15:45:00.730181 20836 consensus_queue.cc:260] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.730460 20837 raft_consensus.cc:348] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.730726 20837 raft_consensus.cc:374] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:00.730824 20837 raft_consensus.cc:729] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbdcc4f879041f8953c4b203a208781, State: Initialized, Role: FOLLOWER
I20260928 15:45:00.731083 20836 ts_tablet_manager.cc:1428] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:45:00.731101 20837 consensus_queue.cc:260] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.733875 20735 heartbeater.cc:503] Master 127.13.5.124:33123 was elected leader, sending a full tablet report...
I20260928 15:45:00.734570 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.731750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.67:34067)
0928 15:45:00.733987 (+ 2237us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.734194 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2059,"thread_start_us":1737,"threads_started":1}
I20260928 15:45:00.736356 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.735922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.67:53081)
0928 15:45:00.736164 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.736248 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 15:45:00.738109 20837 ts_tablet_manager.cc:1428] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20260928 15:45:00.739184 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.738254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.66:51483)
0928 15:45:00.738821 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.738973 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":299,"threads_started":1}
I20260928 15:45:00.740208 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.739752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.66:35793)
0928 15:45:00.739990 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.740078 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 15:45:00.740550 20594 heartbeater.cc:503] Master 127.13.5.124:33123 was elected leader, sending a full tablet report...
I20260928 15:45:00.839380 20841 raft_consensus.cc:482] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:45:00.839654 20841 raft_consensus.cc:504] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.841512 20841 leader_election.cc:288] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295), dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093)
I20260928 15:45:00.847298 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.841783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43093 (local address 127.13.5.65:34919)
0928 15:45:00.842061 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.842080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:00.842223 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:00.842584 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:00.842589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:00.842613 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:00.842886 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.842900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.843981 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.843986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.845376 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.845387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.846495 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.846501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.846696 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.846720 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:45:00.846829 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:45:00.846979 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":21}
I20260928 15:45:00.847267 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.841928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:34919 (local address 127.13.5.66:43093)
0928 15:45:00.842163 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.842169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.842188 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:45:00.842397 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.842401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.842469 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.842567 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:00.843056 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.843815 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.845545 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.846304 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.846470 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.846578 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.846858 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.846992 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:45:00.847350 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.841864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:47325 (local address 127.13.5.67:43295)
0928 15:45:00.842083 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.842089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.842320 (+ 231us) server_negotiation.cc:407] Connection header received
0928 15:45:00.842516 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.842521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.842589 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.842683 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:00.843216 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.843980 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.845632 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.846419 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.846572 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.846680 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.846949 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.847078 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:45:00.847374 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.841783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:43295 (local address 127.13.5.65:47325)
0928 15:45:00.842214 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.842228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:00.842369 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:00.842717 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:00.842722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:00.842745 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:00.843072 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.843081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.844133 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.844139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.845481 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:00.845493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.846624 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.846630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:00.846787 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.846810 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:45:00.846918 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:45:00.847060 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":231}
I20260928 15:45:00.848270 20549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c8770fd18b4fe1883a0426a228f507" candidate_uuid: "5b75b8b58bd4425499d56d72b7158364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbdcc4f879041f8953c4b203a208781" is_pre_election: true
I20260928 15:45:00.848446 20690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c8770fd18b4fe1883a0426a228f507" candidate_uuid: "5b75b8b58bd4425499d56d72b7158364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c830cd92804c4c96bf17a480cf896d6d" is_pre_election: true
I20260928 15:45:00.848644 20549 raft_consensus.cc:2454] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b75b8b58bd4425499d56d72b7158364 in term 0.
I20260928 15:45:00.848791 20690 raft_consensus.cc:2454] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b75b8b58bd4425499d56d72b7158364 in term 0.
I20260928 15:45:00.849304 20339 leader_election.cc:302] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b75b8b58bd4425499d56d72b7158364, dfbdcc4f879041f8953c4b203a208781; no voters:
I20260928 15:45:00.849959 20841 raft_consensus.cc:2781] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:45:00.850106 20841 raft_consensus.cc:482] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:45:00.850209 20841 raft_consensus.cc:3037] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:00.852520 20841 raft_consensus.cc:504] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.853137 20841 leader_election.cc:288] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [CANDIDATE]: Term 1 election: Requested vote from peers c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295), dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093)
I20260928 15:45:00.853709 20690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c8770fd18b4fe1883a0426a228f507" candidate_uuid: "5b75b8b58bd4425499d56d72b7158364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c830cd92804c4c96bf17a480cf896d6d"
I20260928 15:45:00.853709 20549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c8770fd18b4fe1883a0426a228f507" candidate_uuid: "5b75b8b58bd4425499d56d72b7158364" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbdcc4f879041f8953c4b203a208781"
I20260928 15:45:00.853935 20690 raft_consensus.cc:3037] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:00.853945 20549 raft_consensus.cc:3037] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:00.856990 20549 raft_consensus.cc:2454] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b75b8b58bd4425499d56d72b7158364 in term 1.
I20260928 15:45:00.857028 20690 raft_consensus.cc:2454] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b75b8b58bd4425499d56d72b7158364 in term 1.
I20260928 15:45:00.857437 20339 leader_election.cc:302] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b75b8b58bd4425499d56d72b7158364, dfbdcc4f879041f8953c4b203a208781; no voters:
I20260928 15:45:00.857740 20841 raft_consensus.cc:2781] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:00.858173 20841 raft_consensus.cc:686] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 1 LEADER]: Becoming Leader. State: Replica: 5b75b8b58bd4425499d56d72b7158364, State: Running, Role: LEADER
I20260928 15:45:00.858640 20841 consensus_queue.cc:237] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } }
I20260928 15:45:00.859157 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.858732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.65:37157)
0928 15:45:00.858906 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.859016 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 15:45:00.862668 20270 catalog_manager.cc:5495] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65). New cstate: current_term: 1 leader_uuid: "5b75b8b58bd4425499d56d72b7158364" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b75b8b58bd4425499d56d72b7158364" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 } health_report { overall_health: UNKNOWN } } }
I20260928 15:45:00.898895 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.893721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.13.5.65:33255)
0928 15:45:00.893991 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.893996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.894011 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:45:00.894097 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.894101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.894176 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.894335 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:00.894703 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.895434 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.896770 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.897197 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.897336 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.897761 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:00.898309 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:00.898483 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.898544 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.898627 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:45:00.903664 20549 raft_consensus.cc:1261] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Refusing update from remote peer 5b75b8b58bd4425499d56d72b7158364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:00.903810 20690 raft_consensus.cc:1261] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 1 FOLLOWER]: Refusing update from remote peer 5b75b8b58bd4425499d56d72b7158364: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:00.904505 20841 consensus_queue.cc:1035] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfbdcc4f879041f8953c4b203a208781" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:00.904863 20848 consensus_queue.cc:1035] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c830cd92804c4c96bf17a480cf896d6d" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:00.908958 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.907227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.66:60625)
0928 15:45:00.908501 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.908676 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1095}
I20260928 15:45:00.909902 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.908161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.67:60783)
0928 15:45:00.909497 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.909644 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1156}
I20260928 15:45:00.917706 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.916177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.67:41655)
0928 15:45:00.917417 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.917521 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1114}
I20260928 15:45:00.918175 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.916364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.65:39953)
0928 15:45:00.917815 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.917931 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1303}
I20260928 15:45:00.923794 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.923300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.65:39207)
0928 15:45:00.923502 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.923603 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 15:45:00.927374 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.925553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.13.5.66:45055)
0928 15:45:00.926990 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:00.927126 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1249}
I20260928 15:45:00.936530 20269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:00.936837 20269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:00.943614 20269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdfb7623849d4bb1859a7f7f15d2ba90: Sending DeleteTablet for 3 replicas of tablet e9c8770fd18b4fe1883a0426a228f507
I20260928 15:45:00.944694 20384 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c8770fd18b4fe1883a0426a228f507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:00 UTC) from {username='oryx', principal='oryx/127.13.5.124@KRBTEST.COM'} at 127.0.0.1:45506
I20260928 15:45:00.945544 20861 tablet_replica.cc:330] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: stopping tablet replica
I20260928 15:45:00.945791 20861 raft_consensus.cc:2229] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:45:00.945887 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c8770fd18b4fe1883a0426a228f507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:00 UTC) from {username='oryx', principal='oryx/127.13.5.124@KRBTEST.COM'} at 127.0.0.1:59542
I20260928 15:45:00.946013 20528 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c8770fd18b4fe1883a0426a228f507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:00 UTC) from {username='oryx', principal='oryx/127.13.5.124@KRBTEST.COM'} at 127.0.0.1:33312
I20260928 15:45:00.946671 20861 raft_consensus.cc:2258] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:00.948047 20861 ts_tablet_manager.cc:1905] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:00.948594 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 20259
I20260928 15:45:00.960422 20863 tablet_replica.cc:330] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: stopping tablet replica
I20260928 15:45:00.960682 20863 raft_consensus.cc:2229] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:00.961083 20863 raft_consensus.cc:2258] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:00.962518 20863 ts_tablet_manager.cc:1905] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:00.963340 20861 ts_tablet_manager.cc:1918] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:00.963395 20862 tablet_replica.cc:330] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: stopping tablet replica
I20260928 15:45:00.963526 20861 log.cc:1196] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/e9c8770fd18b4fe1883a0426a228f507
I20260928 15:45:00.963616 20862 raft_consensus.cc:2229] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:00.964016 20862 raft_consensus.cc:2258] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:00.964108 20861 ts_tablet_manager.cc:1939] T e9c8770fd18b4fe1883a0426a228f507 P 5b75b8b58bd4425499d56d72b7158364: Deleting consensus metadata
I20260928 15:45:00.965317 20862 ts_tablet_manager.cc:1905] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:00.965551 20254 catalog_manager.cc:4841] TS 5b75b8b58bd4425499d56d72b7158364 (127.13.5.65:33255): tablet e9c8770fd18b4fe1883a0426a228f507 (table test-table [id=084edc2a8dd94618ac9181f59d36c42b]) successfully deleted
I20260928 15:45:00.971406 20863 ts_tablet_manager.cc:1918] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:00.971575 20863 log.cc:1196] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/e9c8770fd18b4fe1883a0426a228f507
I20260928 15:45:00.972144 20863 ts_tablet_manager.cc:1939] T e9c8770fd18b4fe1883a0426a228f507 P dfbdcc4f879041f8953c4b203a208781: Deleting consensus metadata
I20260928 15:45:00.972556 20862 ts_tablet_manager.cc:1918] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:00.972741 20862 log.cc:1196] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/e9c8770fd18b4fe1883a0426a228f507
I20260928 15:45:00.973260 20862 ts_tablet_manager.cc:1939] T e9c8770fd18b4fe1883a0426a228f507 P c830cd92804c4c96bf17a480cf896d6d: Deleting consensus metadata
I20260928 15:45:00.973544 20255 catalog_manager.cc:4841] TS dfbdcc4f879041f8953c4b203a208781 (127.13.5.66:43093): tablet e9c8770fd18b4fe1883a0426a228f507 (table test-table [id=084edc2a8dd94618ac9181f59d36c42b]) successfully deleted
I20260928 15:45:00.974547 20256 catalog_manager.cc:4841] TS c830cd92804c4c96bf17a480cf896d6d (127.13.5.67:43295): tablet e9c8770fd18b4fe1883a0426a228f507 (table test-table [id=084edc2a8dd94618ac9181f59d36c42b]) successfully deleted
I20260928 15:45:00.981606 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:00.949647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.13.5.65:33255)
0928 15:45:00.949837 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:00.949845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:00.958974 (+ 9129us) server_negotiation.cc:407] Connection header received
0928 15:45:00.959351 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:00.959361 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:00.959516 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:00.959629 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:00.966048 (+ 6419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.967449 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.969587 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:00.970305 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:00.970410 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:00.974572 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:00.974587 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:00.974604 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:00.974642 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:00.978173 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:00.979040 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:00.979048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:00.979050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:00.979224 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:00.979646 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:00.979650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:00.979652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:00.980180 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:00.980531 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:00.980913 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 15:45:00.981172 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:45:00.982333 20444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:01.089336 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 20259
I20260928 15:45:01.110433 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 20461
I20260928 15:45:01.125357 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:01.111326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.13.5.66:43093)
0928 15:45:01.111581 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:01.111587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:01.111608 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:45:01.111723 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:01.111729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:01.111805 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:01.112078 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:01.113504 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:01.114407 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:01.115636 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:01.116102 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:01.116242 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:01.118924 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:01.118942 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:01.118961 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:01.118996 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:01.122266 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:01.123080 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:01.123090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:01.123093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:01.123226 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:01.123650 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:01.123657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:01.123660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:01.124131 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:01.124472 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:01.124777 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 15:45:01.125003 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 15:45:01.126197 20589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:01.222857 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 20461
I20260928 15:45:01.243192 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 20607
I20260928 15:45:01.243743 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:01.242496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58556)
0928 15:45:01.243454 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:01.243557 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":691,"threads_started":1}
I20260928 15:45:01.245281 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:01.244943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36553 (local address 127.0.0.1:58562)
0928 15:45:01.245098 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:01.245173 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:36553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:45:01.258035 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:01.244118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.13.5.67:43295)
0928 15:45:01.244339 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:01.244347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:01.244369 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:45:01.244543 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:01.244546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:01.244611 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:01.244693 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:01.246301 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:01.247070 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:01.248709 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:01.249132 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:01.249254 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:01.252123 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:01.252142 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:01.252162 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:01.252211 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:01.255307 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:01.256104 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:01.256111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:01.256114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:01.256260 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:01.256640 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:01.256644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:01.256646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:01.257016 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:01.257238 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:01.257620 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 15:45:01.257799 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:45:01.258756 20730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:01.381515 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 20607
I20260928 15:45:01.404201 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 19943
I20260928 15:45:01.405359 20110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:01.524923 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 19943
I20260928 15:45:01.546514 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 20044
I20260928 15:45:01.547665 20300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:01.672852 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 20044
2026-09-28T15:45:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10209 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:45:01 dist-test-slave-j968 krb5kdc[20874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:45:01 dist-test-slave-j968 krb5kdc[20874](info): set up 2 sockets
Sep 28 15:45:01 dist-test-slave-j968 krb5kdc[20874](info): commencing operation
krb5kdc: starting...
W20260928 15:45:02.843655 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:45:02 dist-test-slave-j968 krb5kdc[20874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:45:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.126.keytab.
Entry for principal oryx/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.126.keytab.
I20260928 15:45:03.071625 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:37011
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.126.keytab
--principal=oryx/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:37011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:03.411334 20887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:03.411703 20887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:03.411913 20887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:03.419315 20887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:45:03.419467 20887 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:45:03.419562 20887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:03.419682 20887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:03.419767 20887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:45:03.419857 20887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:45:03.429884 20887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:37011
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.126.keytab
--principal=oryx/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:03.431272 20887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:03.432677 20887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:03.445089 20892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:03.445122 20895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:03.445153 20893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:03.445852 20887 server_base.cc:1034] running on GCE node
I20260928 15:45:03.446688 20887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:03.448203 20887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:03.449498 20887 hybrid_clock.cc:648] HybridClock initialized: now 1790610303449403 us; error 61 us; skew 500 ppm
I20260928 15:45:03.449874 20887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:03 dist-test-slave-j968 krb5kdc[20874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610303, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:03.459390 20887 init.cc:377] Logged in from keytab as oryx/127.13.5.126@KRBTEST.COM (short username oryx)
I20260928 15:45:03.461016 20887 webserver.cc:460] Webserver started at http://127.13.5.126:37629/ using document root <none> and password file <none>
I20260928 15:45:03.461746 20887 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:03.461838 20887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:03.462116 20887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:03.465278 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae"
format_stamp: "Formatted at 2026-09-28 15:45:03 on dist-test-slave-j968"
server_key: "e0e04df894979459154746c6955d0ba4"
server_key_iv: "88d2357ef24e54bbbf46f15cd68ecfb0"
server_key_version: "encryptionkey@0"
I20260928 15:45:03.466060 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae"
format_stamp: "Formatted at 2026-09-28 15:45:03 on dist-test-slave-j968"
server_key: "e0e04df894979459154746c6955d0ba4"
server_key_iv: "88d2357ef24e54bbbf46f15cd68ecfb0"
server_key_version: "encryptionkey@0"
I20260928 15:45:03.472391 20887 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 15:45:03.477288 20902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:03.478891 20887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 15:45:03.479104 20887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae"
format_stamp: "Formatted at 2026-09-28 15:45:03 on dist-test-slave-j968"
server_key: "e0e04df894979459154746c6955d0ba4"
server_key_iv: "88d2357ef24e54bbbf46f15cd68ecfb0"
server_key_version: "encryptionkey@0"
I20260928 15:45:03.479341 20887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:03.492372 20887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:03.496887 20887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:03.497188 20887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:03.519168 20887 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:37011
I20260928 15:45:03.519187 20963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:37011 every 8 connection(s)
I20260928 15:45:03.520634 20887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:45:03.523797 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 20887
I20260928 15:45:03.524071 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:45:03.524760 13333 external_mini_cluster.cc:1428] Setting key caca67d2bebdbe733f6d6cecbf77218e
I20260928 15:45:03.524955 20964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:03.539579 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: Bootstrap starting.
Sep 28 15:45:03 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:45:03.544082 20964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:03.545713 20964 log.cc:824] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:03.549477 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: No bootstrap required, opened a new log
I20260928 15:45:03.551573 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:03.526844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.13.5.126:37011)
0928 15:45:03.527696 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:03.527711 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:03.527766 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:45:03.530014 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:03.530045 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:03.530596 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:03.531056 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:03.532636 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:03.534606 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:03.535937 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:03.537120 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:03.537315 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:03.541374 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:03.541387 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:03.541409 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:03.541495 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:03.545907 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:03.546920 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:03.546931 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:03.546941 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:03.547194 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:03.547724 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:03.547729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:03.547733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:03.548516 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:03.548817 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:03.549297 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 15:45:03.549779 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":341,"threads_started":1}
I20260928 15:45:03.559530 20964 raft_consensus.cc:348] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37011 } }
I20260928 15:45:03.559964 20964 raft_consensus.cc:374] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:03.560066 20964 raft_consensus.cc:729] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64bdeb07faf47ce9b957c53e1c5d4ae, State: Initialized, Role: FOLLOWER
I20260928 15:45:03.560695 20964 consensus_queue.cc:260] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37011 } }
I20260928 15:45:03.560957 20964 raft_consensus.cc:388] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:45:03.561117 20964 raft_consensus.cc:482] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:45:03.561412 20964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:03.564087 20964 raft_consensus.cc:504] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37011 } }
I20260928 15:45:03.564730 20964 leader_election.cc:302] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a64bdeb07faf47ce9b957c53e1c5d4ae; no voters:
I20260928 15:45:03.565469 20964 leader_election.cc:288] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:45:03.565507 20969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:03.567281 20969 raft_consensus.cc:686] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [term 1 LEADER]: Becoming Leader. State: Replica: a64bdeb07faf47ce9b957c53e1c5d4ae, State: Running, Role: LEADER
I20260928 15:45:03.567988 20964 sys_catalog.cc:564] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:45:03.567952 20969 consensus_queue.cc:237] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37011 } }
I20260928 15:45:03.572052 20971 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a64bdeb07faf47ce9b957c53e1c5d4ae. Latest consensus state: current_term: 1 leader_uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37011 } } }
I20260928 15:45:03.572611 20971 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:03.571725 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64bdeb07faf47ce9b957c53e1c5d4ae" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 37011 } } }
I20260928 15:45:03.573164 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:03.577270 20977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:45:03.585047 20977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:45:03.595943 20977 catalog_manager.cc:1347] Generated new cluster ID: bfaa25b5a9fe4dec8bcd9c7618ca0a11
I20260928 15:45:03.596143 20977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:45:03.611122 20977 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:45:03.612807 20977 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:45:03.621765 20977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: Generated new TSK 0
I20260928 15:45:03.622663 20977 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab.
Entry for principal oryx/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab.
I20260928 15:45:03.700973 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:37011
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab
--principal=oryx/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:04.041976 20990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:04.042266 20990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:04.042433 20990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:04.050103 20990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:04.050271 20990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:04.050385 20990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:45:04.050469 20990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:45:04.060647 20990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab
--principal=oryx/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:37011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:04.062043 20990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:04.063356 20990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:04.077843 20995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:04.077800 20996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:04.078197 20998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:04.078698 20990 server_base.cc:1034] running on GCE node
I20260928 15:45:04.079303 20990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:04.080367 20990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:04.081620 20990 hybrid_clock.cc:648] HybridClock initialized: now 1790610304081551 us; error 34 us; skew 500 ppm
I20260928 15:45:04.082000 20990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:04 dist-test-slave-j968 krb5kdc[20874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610304, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:04.085564 20990 init.cc:377] Logged in from keytab as oryx/127.13.5.65@KRBTEST.COM (short username oryx)
I20260928 15:45:04.087023 20990 webserver.cc:460] Webserver started at http://127.13.5.65:40927/ using document root <none> and password file <none>
I20260928 15:45:04.087761 20990 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:04.087909 20990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:04.088249 20990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:04.091195 20990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "f61fd9e7b9444c55b0966673a222b7bc"
format_stamp: "Formatted at 2026-09-28 15:45:04 on dist-test-slave-j968"
server_key: "9314200831b62e0b15c98932ea91d12b"
server_key_iv: "1c6c39f299b217fe145b4f80c215c752"
server_key_version: "encryptionkey@0"
I20260928 15:45:04.092110 20990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "f61fd9e7b9444c55b0966673a222b7bc"
format_stamp: "Formatted at 2026-09-28 15:45:04 on dist-test-slave-j968"
server_key: "9314200831b62e0b15c98932ea91d12b"
server_key_iv: "1c6c39f299b217fe145b4f80c215c752"
server_key_version: "encryptionkey@0"
I20260928 15:45:04.097476 20990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:45:04.101369 21005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:04.102787 20990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 15:45:04.103001 20990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "f61fd9e7b9444c55b0966673a222b7bc"
format_stamp: "Formatted at 2026-09-28 15:45:04 on dist-test-slave-j968"
server_key: "9314200831b62e0b15c98932ea91d12b"
server_key_iv: "1c6c39f299b217fe145b4f80c215c752"
server_key_version: "encryptionkey@0"
I20260928 15:45:04.103241 20990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:04.137238 20990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:04.143203 20990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:04.143525 20990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:04.146538 20990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:04.146657 20990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:04.146816 20990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:04.146899 20990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:45:04 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610304, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:45:04.175192 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:04.147561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:33761 (local address 127.13.5.126:37011)
0928 15:45:04.148130 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:04.148138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:04.149538 (+ 1400us) server_negotiation.cc:407] Connection header received
0928 15:45:04.152904 (+ 3366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:04.152914 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:04.153044 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:04.153151 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:04.155818 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.156996 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.160235 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.160813 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.160992 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:04.166813 (+ 5821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:04.166822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:04.166829 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:04.166903 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:04.170880 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:04.172109 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:04.172118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:04.172123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:04.172306 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:04.172952 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:04.172958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:04.172968 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:04.173365 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:04.173549 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:04.174532 (+ 983us) server_negotiation.cc:299] Negotiation successful
0928 15:45:04.174776 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":293,"threads_started":1}
I20260928 15:45:04.176452 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:04.148380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37011 (local address 127.13.5.65:33761)
0928 15:45:04.149327 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:04.149380 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:04.152471 (+ 3091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:04.153517 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:04.153534 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:04.154503 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:04.155563 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:04.155595 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.157204 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.157214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:04.160023 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:04.160069 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.161123 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.161135 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:04.161582 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:04.162615 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:04.162665 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:04.166468 (+ 3803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:04.171064 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:04.171071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:04.171090 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:04.171842 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:04.172424 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:04.172431 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:04.172438 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:04.172802 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:04.173619 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:04.173625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:04.174073 (+ 448us) client_negotiation.cc:769] Sending connection context
0928 15:45:04.174423 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 15:45:04.175028 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":391,"threads_started":1}
I20260928 15:45:04.202365 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:04.187538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:50455 (local address 127.13.5.126:37011)
0928 15:45:04.187781 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:04.187790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:04.187818 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:45:04.188081 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:04.188086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:04.188173 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:04.188277 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:04.189246 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.189968 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.191530 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.192346 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.192503 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:04.195148 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:04.195153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:04.195156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:04.195191 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:04.198646 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:04.199666 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:04.199673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:04.199679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:04.199853 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:04.200339 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:04.200344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:04.200347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:04.200806 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:04.200971 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:04.201744 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 15:45:04.201982 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:45:04.202513 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:04.186983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37011 (local address 127.13.5.65:50455)
0928 15:45:04.187724 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:04.187758 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:04.187928 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:04.188305 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:04.188310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:04.188791 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:04.189099 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:04.189109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.190099 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.190105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:04.191308 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:04.191323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.192585 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.192592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:04.192726 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:04.193437 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:04.193471 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:04.194925 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:04.198828 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:04.198836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:04.198842 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:04.199506 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:04.200026 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:04.200033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:04.200042 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:04.200212 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:04.200950 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:04.200954 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:04.201399 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 15:45:04.201647 (+ 248us) client_negotiation.cc:240] Negotiation successful
0928 15:45:04.202042 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":419,"threads_started":1}
I20260928 15:45:04.211120 20990 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:42775
I20260928 15:45:04.211139 21125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:42775 every 8 connection(s)
I20260928 15:45:04.212513 20990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:45:04.214740 21126 heartbeater.cc:347] Connected to a master server at 127.13.5.126:37011
I20260928 15:45:04.215005 21126 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:04.215754 21126 heartbeater.cc:511] Master 127.13.5.126:37011 requested a full tablet report, sending...
I20260928 15:45:04.218065 20919 ts_manager.cc:194] Registered new tserver with Master: f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775)
I20260928 15:45:04.219576 20919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:33761
I20260928 15:45:04.221873 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 20990
I20260928 15:45:04.222039 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:45:04.222692 13333 external_mini_cluster.cc:1428] Setting key b93e0a221b9c04213fe3a318c0bbfb01
WARNING: no policy specified for oryx/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab.
Entry for principal oryx/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab.
I20260928 15:45:04.273615 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:37011
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab
--principal=oryx/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:04.611578 21130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:04.611925 21130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:04.612088 21130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:04.619496 21130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:04.619613 21130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:04.619757 21130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:45:04.619845 21130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:45:04.630754 21130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab
--principal=oryx/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:37011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:04.632244 21130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:04.633646 21130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:04.647471 21136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:04.647516 21138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:04.647562 21135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:04.648555 21130 server_base.cc:1034] running on GCE node
I20260928 15:45:04.649247 21130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:04.650310 21130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:04.651608 21130 hybrid_clock.cc:648] HybridClock initialized: now 1790610304651577 us; error 123 us; skew 500 ppm
I20260928 15:45:04.652032 21130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:04 dist-test-slave-j968 krb5kdc[20874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610304, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:04.655337 21130 init.cc:377] Logged in from keytab as oryx/127.13.5.66@KRBTEST.COM (short username oryx)
I20260928 15:45:04.656972 21130 webserver.cc:460] Webserver started at http://127.13.5.66:46381/ using document root <none> and password file <none>
I20260928 15:45:04.657714 21130 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:04.657850 21130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:04.658209 21130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:04.661319 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "3b90f9e4e2dd48b9b914eaba542405db"
format_stamp: "Formatted at 2026-09-28 15:45:04 on dist-test-slave-j968"
server_key: "b7ea80b10143ac71fb40ade78d57da32"
server_key_iv: "8a9feffa489df3c663e3e3a11bc26aaf"
server_key_version: "encryptionkey@0"
I20260928 15:45:04.662175 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "3b90f9e4e2dd48b9b914eaba542405db"
format_stamp: "Formatted at 2026-09-28 15:45:04 on dist-test-slave-j968"
server_key: "b7ea80b10143ac71fb40ade78d57da32"
server_key_iv: "8a9feffa489df3c663e3e3a11bc26aaf"
server_key_version: "encryptionkey@0"
I20260928 15:45:04.667820 21130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:45:04.671810 21145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:04.673532 21130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:04.673722 21130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "3b90f9e4e2dd48b9b914eaba542405db"
format_stamp: "Formatted at 2026-09-28 15:45:04 on dist-test-slave-j968"
server_key: "b7ea80b10143ac71fb40ade78d57da32"
server_key_iv: "8a9feffa489df3c663e3e3a11bc26aaf"
server_key_version: "encryptionkey@0"
I20260928 15:45:04.673990 21130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:04.689493 21130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:04.693540 21130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:04.693819 21130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:04.696558 21130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:04.696656 21130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:04.696784 21130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:04.696861 21130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:45:04 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610304, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:45:04.725214 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:04.697836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37011 (local address 127.13.5.66:56747)
0928 15:45:04.698711 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:04.698756 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:04.700762 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:04.701774 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:04.701786 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:04.702806 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:04.704004 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:04.704037 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.705979 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.705988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:04.709020 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:04.709045 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.710215 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.710235 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:04.710468 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:04.711260 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:04.711302 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:04.715080 (+ 3778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:04.719265 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:04.719279 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:04.719304 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:04.720058 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:04.720745 (+ 687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:04.720753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:04.720757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:04.721092 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:04.721878 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:04.721887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:04.722469 (+ 582us) client_negotiation.cc:769] Sending connection context
0928 15:45:04.722819 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 15:45:04.723482 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":398,"threads_started":1}
I20260928 15:45:04.726341 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:04.697209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:56747 (local address 127.13.5.126:37011)
0928 15:45:04.697425 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:04.697432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:04.698887 (+ 1455us) server_negotiation.cc:407] Connection header received
0928 15:45:04.701054 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:04.701060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:04.701280 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:04.701413 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:04.704277 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.705628 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.709219 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.709913 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.710068 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:04.715336 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:04.715341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:04.715344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:04.715382 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:04.719060 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:04.720230 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:04.720244 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:04.720246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:04.720366 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:04.721246 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:04.721253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:04.721258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:04.721704 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:04.721899 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:04.725809 (+ 3910us) server_negotiation.cc:299] Negotiation successful
0928 15:45:04.726017 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:45:04.748347 21130 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:46741
I20260928 15:45:04.748399 21264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:46741 every 8 connection(s)
I20260928 15:45:04.748865 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:04.731692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:44745 (local address 127.13.5.126:37011)
0928 15:45:04.731954 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:04.731959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:04.732804 (+ 845us) server_negotiation.cc:407] Connection header received
0928 15:45:04.733057 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:04.733065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:04.733165 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:04.733276 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:04.734742 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.735474 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.738112 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.738754 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.738914 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:04.741394 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:04.741402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:04.741409 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:04.741459 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:04.745160 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:04.746323 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:04.746330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:04.746336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:04.746519 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:04.746951 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:04.746956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:04.746959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:04.747553 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:04.747754 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:04.748345 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 15:45:04.748616 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:45:04.749040 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:04.732033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37011 (local address 127.13.5.66:44745)
0928 15:45:04.732683 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:04.732699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:04.732890 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:04.733342 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:04.733350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:04.734073 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:04.734549 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:04.734566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.735617 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.735625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:04.737928 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:04.737954 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:04.738977 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:04.738985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:04.739164 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:04.739932 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:04.739993 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:04.741138 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:04.745338 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:04.745346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:04.745352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:04.746157 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:04.746650 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:04.746655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:04.746658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:04.746826 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:04.747687 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:04.747695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:04.747977 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 15:45:04.748198 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 15:45:04.748617 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":376,"threads_started":1}
I20260928 15:45:04.750093 21130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:45:04.752718 21265 heartbeater.cc:347] Connected to a master server at 127.13.5.126:37011
I20260928 15:45:04.753000 21265 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:04.753703 21265 heartbeater.cc:511] Master 127.13.5.126:37011 requested a full tablet report, sending...
I20260928 15:45:04.755069 20919 ts_manager.cc:194] Registered new tserver with Master: 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741)
I20260928 15:45:04.755923 20919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:56747
I20260928 15:45:04.757066 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 21130
I20260928 15:45:04.757234 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:45:04.757797 13333 external_mini_cluster.cc:1428] Setting key 9dc0aa9b2b69865bd16a87cda77df018
WARNING: no policy specified for oryx/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab.
Entry for principal oryx/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab.
I20260928 15:45:04.808722 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:37011
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab
--principal=oryx/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:05.142925 21269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:05.143265 21269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:05.143452 21269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:05.150983 21269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:05.151155 21269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:05.151327 21269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:45:05.151484 21269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:45:05.161770 21269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab
--principal=oryx/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.5.126:37011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:05.163152 21269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:05.164621 21269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:05.178642 21274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:05.178673 21275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:05.178555 21277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:05.179136 21269 server_base.cc:1034] running on GCE node
I20260928 15:45:05.180209 21269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:05.181227 21269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:05.182461 21269 hybrid_clock.cc:648] HybridClock initialized: now 1790610305182381 us; error 57 us; skew 500 ppm
I20260928 15:45:05.182794 21269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:05 dist-test-slave-j968 krb5kdc[20874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610305, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:05.186259 21269 init.cc:377] Logged in from keytab as oryx/127.13.5.67@KRBTEST.COM (short username oryx)
I20260928 15:45:05.187836 21269 webserver.cc:460] Webserver started at http://127.13.5.67:35913/ using document root <none> and password file <none>
I20260928 15:45:05.188637 21269 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:05.188776 21269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:05.189141 21269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:05.192240 21269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "f2fad6c82e1844ac9c9922653d61d922"
format_stamp: "Formatted at 2026-09-28 15:45:05 on dist-test-slave-j968"
server_key: "4e97dcea2b9c9e688671656478ab62b4"
server_key_iv: "3b4d95a82650fb48a4211a076dc4d570"
server_key_version: "encryptionkey@0"
I20260928 15:45:05.193053 21269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "f2fad6c82e1844ac9c9922653d61d922"
format_stamp: "Formatted at 2026-09-28 15:45:05 on dist-test-slave-j968"
server_key: "4e97dcea2b9c9e688671656478ab62b4"
server_key_iv: "3b4d95a82650fb48a4211a076dc4d570"
server_key_version: "encryptionkey@0"
I20260928 15:45:05.198339 21269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:45:05.202030 21284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:05.203405 21269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 15:45:05.203580 21269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "f2fad6c82e1844ac9c9922653d61d922"
format_stamp: "Formatted at 2026-09-28 15:45:05 on dist-test-slave-j968"
server_key: "4e97dcea2b9c9e688671656478ab62b4"
server_key_iv: "3b4d95a82650fb48a4211a076dc4d570"
server_key_version: "encryptionkey@0"
I20260928 15:45:05.203814 21269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:05.222908 21126 heartbeater.cc:503] Master 127.13.5.126:37011 was elected leader, sending a full tablet report...
I20260928 15:45:05.226981 21269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:05.231554 21269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:05.231779 21269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:05.234481 21269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:05.234601 21269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:05.234767 21269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:05.234845 21269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:45:05 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610305, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:45:05.262977 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.235349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:49385 (local address 127.13.5.126:37011)
0928 15:45:05.235539 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.235545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.237168 (+ 1623us) server_negotiation.cc:407] Connection header received
0928 15:45:05.239968 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.239977 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.240102 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.240223 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:05.243007 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.244384 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.247493 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.248309 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.248486 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.254387 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:05.254396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:05.254403 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:05.254465 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:05.258026 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:05.259278 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:05.259287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:05.259293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:05.259484 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:05.260219 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:05.260228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:05.260231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:05.260748 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:05.260941 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.262130 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.262468 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:45:05.264804 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.236061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37011 (local address 127.13.5.67:49385)
0928 15:45:05.236973 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.237029 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.239548 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.240645 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.240659 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.241694 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:05.242745 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.242781 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.244582 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.244592 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.247271 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.247293 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.248638 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.248649 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.248927 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.249881 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:05.249930 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:05.254021 (+ 4091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:05.258209 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:05.258217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:05.258264 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:05.259087 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:05.259654 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:05.259665 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:05.259670 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:05.260083 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:05.260895 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:05.260903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:05.261539 (+ 636us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.261982 (+ 443us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.262714 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":405,"threads_started":1}
I20260928 15:45:05.279277 21269 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:39657
I20260928 15:45:05.279299 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:39657 every 8 connection(s)
I20260928 15:45:05.280795 21269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:45:05.281301 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 21269
I20260928 15:45:05.281525 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:45:05.282260 13333 external_mini_cluster.cc:1428] Setting key 64bdf6c001b6b442ac5b4f4e5281489e
I20260928 15:45:05.284343 21404 heartbeater.cc:347] Connected to a master server at 127.13.5.126:37011
I20260928 15:45:05.284669 21404 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:05.285652 21404 heartbeater.cc:511] Master 127.13.5.126:37011 requested a full tablet report, sending...
I20260928 15:45:05.287760 20918 ts_manager.cc:194] Registered new tserver with Master: f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657)
I20260928 15:45:05.288800 20918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:49385
I20260928 15:45:05.292152 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.273303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:53471 (local address 127.13.5.126:37011)
0928 15:45:05.273554 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.273563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.276930 (+ 3367us) server_negotiation.cc:407] Connection header received
0928 15:45:05.277172 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.277177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.277259 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.277379 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:05.278684 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.279753 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.281216 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.281888 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.282016 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.285039 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:05.285047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:05.285054 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:05.285100 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:05.289260 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:05.290151 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:05.290155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:05.290157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:05.290284 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:05.290722 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:05.290726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:05.290728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:05.291006 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:05.291137 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.291643 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.291814 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:45:05.292182 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.273269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37011 (local address 127.13.5.67:53471)
0928 15:45:05.276778 (+ 3509us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.276801 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.277010 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.277400 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.277405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.278103 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:05.278530 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.278545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.279929 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.279935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.281061 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.281082 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.282276 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.282283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.282498 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.283350 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:05.283422 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:05.284763 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:05.289394 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:05.289397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:05.289400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:05.290002 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:05.290396 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:05.290401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:05.290405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:05.290571 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:05.291136 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:05.291141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:05.291319 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.291479 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.291809 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3337,"thread_start_us":3107,"threads_started":1}
I20260928 15:45:05.298979 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:45:05 dist-test-slave-j968 krb5kdc[20874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:45:05 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:05 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:45:05.350672 21014 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:05.339316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.13.5.126:37011)
0928 15:45:05.339585 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.339591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.339779 (+ 188us) server_negotiation.cc:407] Connection header received
0928 15:45:05.340017 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.340021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.340094 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.340183 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:05.341562 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.342331 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.343537 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.344154 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.344265 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.350318 (+ 6053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32992: BlockingRecv error: recv got EOF from 127.0.0.1:32992 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 28 15:45:05 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:45:05.367990 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.354466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.13.5.126:37011)
0928 15:45:05.354698 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.354704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.354900 (+ 196us) server_negotiation.cc:407] Connection header received
0928 15:45:05.355105 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.355117 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.355195 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.355302 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:05.356517 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.357258 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.358511 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.358906 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.359048 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.362853 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:05.362858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:05.362861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:05.362898 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:05.365465 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:05.366127 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:05.366130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:05.366132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:05.366247 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:05.366564 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:05.366568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:05.366569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:05.366847 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:05.367107 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.367469 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.367688 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:45:05.372208 20918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:45:05.379426 20918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:45:05.405707 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.392332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.13.5.65:42775)
0928 15:45:05.398259 (+ 5927us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.398264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.398286 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:45:05.398361 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.398366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.398536 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.399042 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:05.399588 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.401097 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.403169 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.404572 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.404712 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.404941 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.405210 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.405375 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5782,"thread_start_us":5642,"threads_started":1}
I20260928 15:45:05.405833 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.392322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42775 (local address 127.0.0.1:54592)
0928 15:45:05.394145 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.394190 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.394349 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.398778 (+ 4429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.398782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.398804 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:05.399343 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.399361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.401296 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.401302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.403008 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.403020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.404768 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.404774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.404976 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.405026 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.405166 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.405375 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1561,"mutex_wait_us":54,"spinlock_wait_cycles":22784,"thread_start_us":2820,"threads_started":1}
I20260928 15:45:05.406484 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.392345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39657 (local address 127.0.0.1:58828)
0928 15:45:05.394930 (+ 2585us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.394948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.395057 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.398180 (+ 3123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.398187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.398216 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:05.398609 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.398624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.401168 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.401172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.403603 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.403618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.405449 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.405456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.405631 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.405680 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.405744 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.406048 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2458,"mutex_wait_us":52,"thread_start_us":1427,"threads_started":1}
I20260928 15:45:05.406507 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.394748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:46741 (local address 127.0.0.1:52242)
0928 15:45:05.397823 (+ 3075us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.397841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.398003 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.398680 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.398684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.398715 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:05.399263 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.399290 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.400941 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.400948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.403612 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.403620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.405693 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.405699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.405860 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.405917 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.405996 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.406093 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2935,"mutex_wait_us":93,"thread_start_us":729,"threads_started":1}
I20260928 15:45:05.406792 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.395248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.13.5.66:46741)
0928 15:45:05.397764 (+ 2516us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.397777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.398135 (+ 358us) server_negotiation.cc:407] Connection header received
0928 15:45:05.398229 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.398235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.398460 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.398664 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:05.399514 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.400793 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.403794 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.405206 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.405341 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.405549 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.406362 (+ 813us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.406542 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2308,"thread_start_us":2162,"threads_started":1}
I20260928 15:45:05.407954 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.392385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.13.5.67:39657)
0928 15:45:05.397468 (+ 5083us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.397487 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.397520 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:45:05.397620 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.397629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.397925 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.398818 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:05.399156 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.401019 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.403796 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.405206 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.407006 (+ 1800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.407307 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.407379 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.407606 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4868,"thread_start_us":4646,"threads_started":1}
I20260928 15:45:05.410145 21056 tablet_service.cc:1467] Processing CreateTablet for tablet 9350d903d3914b4a8ca751d08a3ff3a6 (DEFAULT_TABLE table=test-table [id=e0d64038d0404a27b947ac7427f3273b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:05.410308 21195 tablet_service.cc:1467] Processing CreateTablet for tablet 9350d903d3914b4a8ca751d08a3ff3a6 (DEFAULT_TABLE table=test-table [id=e0d64038d0404a27b947ac7427f3273b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:05.411389 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350d903d3914b4a8ca751d08a3ff3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.411406 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350d903d3914b4a8ca751d08a3ff3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.415936 21334 tablet_service.cc:1467] Processing CreateTablet for tablet 9350d903d3914b4a8ca751d08a3ff3a6 (DEFAULT_TABLE table=test-table [id=e0d64038d0404a27b947ac7427f3273b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:05.417438 21334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9350d903d3914b4a8ca751d08a3ff3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.423623 21428 tablet_bootstrap.cc:492] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Bootstrap starting.
I20260928 15:45:05.425685 21429 tablet_bootstrap.cc:492] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Bootstrap starting.
I20260928 15:45:05.427448 21428 tablet_bootstrap.cc:654] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.428818 21428 log.cc:824] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:05.429257 21429 tablet_bootstrap.cc:654] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.430445 21429 log.cc:824] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:05.432538 21428 tablet_bootstrap.cc:492] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: No bootstrap required, opened a new log
I20260928 15:45:05.432886 21428 ts_tablet_manager.cc:1397] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 15:45:05.436640 21432 tablet_bootstrap.cc:492] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Bootstrap starting.
I20260928 15:45:05.439744 21429 tablet_bootstrap.cc:492] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: No bootstrap required, opened a new log
I20260928 15:45:05.440075 21429 ts_tablet_manager.cc:1397] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260928 15:45:05.440284 21432 tablet_bootstrap.cc:654] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.441560 21432 log.cc:824] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:05.442545 21428 raft_consensus.cc:348] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.443010 21428 raft_consensus.cc:374] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.443115 21428 raft_consensus.cc:729] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b90f9e4e2dd48b9b914eaba542405db, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.443765 21428 consensus_queue.cc:260] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.446153 21432 tablet_bootstrap.cc:492] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: No bootstrap required, opened a new log
I20260928 15:45:05.446398 21432 ts_tablet_manager.cc:1397] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 15:45:05.447338 21265 heartbeater.cc:503] Master 127.13.5.126:37011 was elected leader, sending a full tablet report...
I20260928 15:45:05.448031 21428 ts_tablet_manager.cc:1428] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260928 15:45:05.451634 21429 raft_consensus.cc:348] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.452165 21429 raft_consensus.cc:374] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.452270 21429 raft_consensus.cc:729] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61fd9e7b9444c55b0966673a222b7bc, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.452894 21429 consensus_queue.cc:260] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.454726 21429 ts_tablet_manager.cc:1428] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260928 15:45:05.456760 21432 raft_consensus.cc:348] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.457134 21432 raft_consensus.cc:374] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.457245 21432 raft_consensus.cc:729] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad6c82e1844ac9c9922653d61d922, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.457896 21432 consensus_queue.cc:260] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.459021 21404 heartbeater.cc:503] Master 127.13.5.126:37011 was elected leader, sending a full tablet report...
I20260928 15:45:05.459455 21432 ts_tablet_manager.cc:1428] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260928 15:45:05.464918 21127 tablet.cc:2369] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:45:05.466126 21436 raft_consensus.cc:482] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:45:05.466378 21436 raft_consensus.cc:504] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.468098 21436 leader_election.cc:288] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741)
I20260928 15:45:05.475175 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.468370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:59631 (local address 127.13.5.65:42775)
0928 15:45:05.468916 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.468925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.468951 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:45:05.469051 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.469056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.469132 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.469239 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:05.469718 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.470744 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.473149 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.474259 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.474401 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.474511 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.474785 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.474927 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260928 15:45:05.475217 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.468294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42775 (local address 127.13.5.67:59631)
0928 15:45:05.468593 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.468615 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.468785 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.469273 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.469280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.469314 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:05.469561 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.469572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.470889 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.470893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.472974 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.472994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.474472 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.474486 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.474623 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.474649 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.474728 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.474825 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":42}
I20260928 15:45:05.475909 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.468646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:34339 (local address 127.13.5.66:46741)
0928 15:45:05.468874 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.468882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.469413 (+ 531us) server_negotiation.cc:407] Connection header received
0928 15:45:05.469621 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.469643 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.469722 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.469827 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:05.470292 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.471263 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.473602 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.474757 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.474895 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.474993 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.475517 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.475654 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:45:05.476008 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.468569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:46741 (local address 127.13.5.67:34339)
0928 15:45:05.469309 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.469325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.469442 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.469832 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.469835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.469855 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:05.470129 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.470136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.471404 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.471409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.473398 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.473408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.475007 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.475013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.475181 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.475224 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.475460 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.475594 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":25,"thread_start_us":434,"threads_started":1}
I20260928 15:45:05.476399 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350d903d3914b4a8ca751d08a3ff3a6" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc" is_pre_election: true
I20260928 15:45:05.476979 21080 raft_consensus.cc:2454] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 0.
I20260928 15:45:05.477018 21215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350d903d3914b4a8ca751d08a3ff3a6" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b90f9e4e2dd48b9b914eaba542405db" is_pre_election: true
I20260928 15:45:05.477535 21215 raft_consensus.cc:2454] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 0.
I20260928 15:45:05.477661 21287 leader_election.cc:302] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad6c82e1844ac9c9922653d61d922, f61fd9e7b9444c55b0966673a222b7bc; no voters:
I20260928 15:45:05.478107 21436 raft_consensus.cc:2781] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:45:05.478262 21436 raft_consensus.cc:482] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:45:05.478379 21436 raft_consensus.cc:3037] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.480840 21436 raft_consensus.cc:504] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.481469 21436 leader_election.cc:288] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 election: Requested vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741)
I20260928 15:45:05.482041 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350d903d3914b4a8ca751d08a3ff3a6" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc"
I20260928 15:45:05.482141 21215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9350d903d3914b4a8ca751d08a3ff3a6" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b90f9e4e2dd48b9b914eaba542405db"
I20260928 15:45:05.482273 21080 raft_consensus.cc:3037] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.482367 21215 raft_consensus.cc:3037] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.484758 21080 raft_consensus.cc:2454] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 1.
I20260928 15:45:05.484910 21215 raft_consensus.cc:2454] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 1.
I20260928 15:45:05.485177 21287 leader_election.cc:302] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad6c82e1844ac9c9922653d61d922, f61fd9e7b9444c55b0966673a222b7bc; no voters:
I20260928 15:45:05.485477 21436 raft_consensus.cc:2781] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:05.485956 21436 raft_consensus.cc:686] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 1 LEADER]: Becoming Leader. State: Replica: f2fad6c82e1844ac9c9922653d61d922, State: Running, Role: LEADER
I20260928 15:45:05.486503 21436 consensus_queue.cc:237] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } }
I20260928 15:45:05.491683 20918 catalog_manager.cc:5495] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } health_report { overall_health: UNKNOWN } } }
W20260928 15:45:05.502807 21266 tablet.cc:2369] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:45:05.530198 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.523946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.13.5.67:39657)
0928 15:45:05.524133 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.524138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.524156 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:45:05.524327 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.524331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.524475 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.524608 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:05.524992 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.525744 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.527717 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.528201 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.528324 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.528881 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:05.529587 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:05.529712 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.529835 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.529908 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 15:45:05.537698 21405 tablet.cc:2369] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:45:05.539677 21215 raft_consensus.cc:1261] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Refusing update from remote peer f2fad6c82e1844ac9c9922653d61d922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:05.539695 21080 raft_consensus.cc:1261] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Refusing update from remote peer f2fad6c82e1844ac9c9922653d61d922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:05.540762 21438 consensus_queue.cc:1035] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:05.541126 21436 consensus_queue.cc:1035] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:05.566303 20918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:05.566540 20918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:05.570549 20918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: Sending DeleteTablet for 3 replicas of tablet 9350d903d3914b4a8ca751d08a3ff3a6
I20260928 15:45:05.571890 21334 tablet_service.cc:1514] Processing DeleteTablet for tablet 9350d903d3914b4a8ca751d08a3ff3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:05 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:58828
I20260928 15:45:05.571952 21056 tablet_service.cc:1514] Processing DeleteTablet for tablet 9350d903d3914b4a8ca751d08a3ff3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:05 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:54592
I20260928 15:45:05.572926 21451 tablet_replica.cc:330] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: stopping tablet replica
I20260928 15:45:05.572803 21195 tablet_service.cc:1514] Processing DeleteTablet for tablet 9350d903d3914b4a8ca751d08a3ff3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:05 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52242
I20260928 15:45:05.573812 21451 raft_consensus.cc:2229] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:45:05.573782 20918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 15:45:05.574496 21451 raft_consensus.cc:2258] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 15:45:05.574565 20918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:45:05.577116 21451 ts_tablet_manager.cc:1905] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:05.583549 21055 tablet_service.cc:1467] Processing CreateTablet for tablet f5121e2011a1476baeb92e3407e36de2 (DEFAULT_TABLE table=test-table [id=9716a50555cb4616af1990a712a2594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:05.584192 21453 tablet_replica.cc:330] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: stopping tablet replica
I20260928 15:45:05.584259 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5121e2011a1476baeb92e3407e36de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.585855 21451 ts_tablet_manager.cc:1918] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:05.586488 21453 raft_consensus.cc:2229] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:05.587131 21453 raft_consensus.cc:2258] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:05.587756 21451 log.cc:1196] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/9350d903d3914b4a8ca751d08a3ff3a6
I20260928 15:45:05.588451 21452 tablet_replica.cc:330] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: stopping tablet replica
I20260928 15:45:05.588519 21195 tablet_service.cc:1467] Processing CreateTablet for tablet f5121e2011a1476baeb92e3407e36de2 (DEFAULT_TABLE table=test-table [id=9716a50555cb4616af1990a712a2594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:05.588907 21452 raft_consensus.cc:2229] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:05.588994 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5121e2011a1476baeb92e3407e36de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.589355 21452 raft_consensus.cc:2258] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:05.590005 21453 ts_tablet_manager.cc:1905] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:05.592377 21334 tablet_service.cc:1467] Processing CreateTablet for tablet f5121e2011a1476baeb92e3407e36de2 (DEFAULT_TABLE table=test-table [id=9716a50555cb4616af1990a712a2594a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:05.592861 21334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5121e2011a1476baeb92e3407e36de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.594058 21452 ts_tablet_manager.cc:1905] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:05.596891 21453 ts_tablet_manager.cc:1918] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:05.597045 21453 log.cc:1196] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/9350d903d3914b4a8ca751d08a3ff3a6
I20260928 15:45:05.597649 21453 ts_tablet_manager.cc:1939] T 9350d903d3914b4a8ca751d08a3ff3a6 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting consensus metadata
I20260928 15:45:05.599731 21451 ts_tablet_manager.cc:1939] T 9350d903d3914b4a8ca751d08a3ff3a6 P f2fad6c82e1844ac9c9922653d61d922: Deleting consensus metadata
I20260928 15:45:05.599696 20906 catalog_manager.cc:4841] TS 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741): tablet 9350d903d3914b4a8ca751d08a3ff3a6 (table test-table [id=e0d64038d0404a27b947ac7427f3273b]) successfully deleted
I20260928 15:45:05.601958 20906 catalog_manager.cc:4841] TS f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657): tablet 9350d903d3914b4a8ca751d08a3ff3a6 (table test-table [id=e0d64038d0404a27b947ac7427f3273b]) successfully deleted
I20260928 15:45:05.602308 21452 ts_tablet_manager.cc:1918] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:05.602497 21452 log.cc:1196] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/9350d903d3914b4a8ca751d08a3ff3a6
I20260928 15:45:05.603116 21452 ts_tablet_manager.cc:1939] T 9350d903d3914b4a8ca751d08a3ff3a6 P f61fd9e7b9444c55b0966673a222b7bc: Deleting consensus metadata
I20260928 15:45:05.605052 20905 catalog_manager.cc:4841] TS f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775): tablet 9350d903d3914b4a8ca751d08a3ff3a6 (table test-table [id=e0d64038d0404a27b947ac7427f3273b]) successfully deleted
I20260928 15:45:05.606031 21428 tablet_bootstrap.cc:492] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: Bootstrap starting.
I20260928 15:45:05.606784 21429 tablet_bootstrap.cc:492] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: Bootstrap starting.
I20260928 15:45:05.607333 21432 tablet_bootstrap.cc:492] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: Bootstrap starting.
I20260928 15:45:05.607805 21428 tablet_bootstrap.cc:654] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.608407 21429 tablet_bootstrap.cc:654] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.609153 21432 tablet_bootstrap.cc:654] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.617805 21428 tablet_bootstrap.cc:492] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: No bootstrap required, opened a new log
I20260928 15:45:05.618017 21428 ts_tablet_manager.cc:1397] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260928 15:45:05.618559 21429 tablet_bootstrap.cc:492] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: No bootstrap required, opened a new log
I20260928 15:45:05.618796 21429 ts_tablet_manager.cc:1397] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 15:45:05.619123 21428 raft_consensus.cc:348] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.619436 21428 raft_consensus.cc:374] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.619550 21428 raft_consensus.cc:729] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b90f9e4e2dd48b9b914eaba542405db, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.619894 21429 raft_consensus.cc:348] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.620194 21429 raft_consensus.cc:374] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.620031 21428 consensus_queue.cc:260] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.620518 21429 raft_consensus.cc:729] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61fd9e7b9444c55b0966673a222b7bc, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.621876 21429 consensus_queue.cc:260] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.623492 21429 ts_tablet_manager.cc:1428] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260928 15:45:05.624367 21428 ts_tablet_manager.cc:1428] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260928 15:45:05.628360 21432 tablet_bootstrap.cc:492] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: No bootstrap required, opened a new log
I20260928 15:45:05.628536 21432 ts_tablet_manager.cc:1397] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.019s
I20260928 15:45:05.629312 21432 raft_consensus.cc:348] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.629484 21432 raft_consensus.cc:374] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.629616 21432 raft_consensus.cc:729] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad6c82e1844ac9c9922653d61d922, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.629895 21432 consensus_queue.cc:260] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.630478 21432 ts_tablet_manager.cc:1428] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 15:45:05.661363 21434 raft_consensus.cc:482] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:45:05.661638 21434 raft_consensus.cc:504] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.663481 21434 leader_election.cc:288] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657)
I20260928 15:45:05.672087 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.664639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:48547 (local address 127.13.5.67:39657)
0928 15:45:05.664825 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.664833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.665317 (+ 484us) server_negotiation.cc:407] Connection header received
0928 15:45:05.665566 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.665572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.665657 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.665761 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:05.666524 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.667670 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.669808 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.670930 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.671078 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.671182 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.671555 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.671719 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:45:05.672138 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.664208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39657 (local address 127.13.5.66:48547)
0928 15:45:05.665225 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.665242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.665402 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.665786 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.665791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.665823 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:05.666334 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.666347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.667797 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.667804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.669643 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.669657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.671164 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.671171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.671358 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.671396 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.671502 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.671661 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":35,"thread_start_us":402,"threads_started":1}
I20260928 15:45:05.672797 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.664228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:58019 (local address 127.13.5.65:42775)
0928 15:45:05.664415 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.664421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.665054 (+ 633us) server_negotiation.cc:407] Connection header received
0928 15:45:05.665361 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.665367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.665472 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.665577 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:05.666263 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.667423 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.670332 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.671520 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.671649 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.671747 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.672347 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.672527 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:45:05.673530 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.663698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42775 (local address 127.13.5.66:58019)
0928 15:45:05.664931 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.664952 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.665178 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.665595 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.665600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.665637 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:05.666095 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.666106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.667568 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.667574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.670158 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.670173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.671940 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.671946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.672146 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.672183 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.673002 (+ 819us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.673145 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":119,"thread_start_us":1221,"threads_started":1}
I20260928 15:45:05.673465 21354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5121e2011a1476baeb92e3407e36de2" candidate_uuid: "3b90f9e4e2dd48b9b914eaba542405db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad6c82e1844ac9c9922653d61d922" is_pre_election: true
I20260928 15:45:05.674023 21354 raft_consensus.cc:2454] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b90f9e4e2dd48b9b914eaba542405db in term 0.
I20260928 15:45:05.674633 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5121e2011a1476baeb92e3407e36de2" candidate_uuid: "3b90f9e4e2dd48b9b914eaba542405db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc" is_pre_election: true
I20260928 15:45:05.674749 21149 leader_election.cc:302] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b90f9e4e2dd48b9b914eaba542405db, f2fad6c82e1844ac9c9922653d61d922; no voters:
I20260928 15:45:05.674893 21080 raft_consensus.cc:2454] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b90f9e4e2dd48b9b914eaba542405db in term 0.
I20260928 15:45:05.675192 21434 raft_consensus.cc:2781] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:45:05.675323 21434 raft_consensus.cc:482] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:45:05.675416 21434 raft_consensus.cc:3037] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.678728 21434 raft_consensus.cc:504] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.679515 21434 leader_election.cc:288] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [CANDIDATE]: Term 1 election: Requested vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657)
I20260928 15:45:05.680069 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5121e2011a1476baeb92e3407e36de2" candidate_uuid: "3b90f9e4e2dd48b9b914eaba542405db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc"
I20260928 15:45:05.680296 21080 raft_consensus.cc:3037] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.682576 21080 raft_consensus.cc:2454] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b90f9e4e2dd48b9b914eaba542405db in term 1.
I20260928 15:45:05.683138 21148 leader_election.cc:302] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b90f9e4e2dd48b9b914eaba542405db, f61fd9e7b9444c55b0966673a222b7bc; no voters:
I20260928 15:45:05.683259 21354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5121e2011a1476baeb92e3407e36de2" candidate_uuid: "3b90f9e4e2dd48b9b914eaba542405db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad6c82e1844ac9c9922653d61d922"
I20260928 15:45:05.683501 21354 raft_consensus.cc:3037] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.683522 21434 raft_consensus.cc:2781] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:05.684396 21434 raft_consensus.cc:686] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 LEADER]: Becoming Leader. State: Replica: 3b90f9e4e2dd48b9b914eaba542405db, State: Running, Role: LEADER
I20260928 15:45:05.684856 21434 consensus_queue.cc:237] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.686578 21354 raft_consensus.cc:2454] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b90f9e4e2dd48b9b914eaba542405db in term 1.
I20260928 15:45:05.689136 20919 catalog_manager.cc:5495] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66). New cstate: current_term: 1 leader_uuid: "3b90f9e4e2dd48b9b914eaba542405db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:45:05 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610303, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.126@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:45:05.738992 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.722814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:37011 (local address 127.0.0.1:33004)
0928 15:45:05.723635 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.723658 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.723794 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.724242 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.724246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.725018 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:05.725288 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.725297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.726768 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.726772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.727730 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.727741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.728807 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.728812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.728932 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.729827 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:05.729844 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:05.732335 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:05.736364 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:05.736377 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:05.736383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:05.736842 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:05.737251 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:05.737254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:05.737256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:05.737397 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:05.738075 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:05.738082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:05.738249 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.738389 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.738631 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":435,"threads_started":1}
I20260928 15:45:05.739198 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.722817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.13.5.126:37011)
0928 15:45:05.723055 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.723062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.723756 (+ 694us) server_negotiation.cc:407] Connection header received
0928 15:45:05.723943 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.723949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.724037 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.724167 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:05.725450 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.726626 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.727905 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.728605 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.728769 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.732535 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:05.732543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:05.732548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:05.732587 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:05.736192 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:05.736972 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:05.736978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:05.736982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:05.737132 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:05.737505 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:05.737510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:05.737514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:05.737927 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:05.738072 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.738537 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.738774 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 15:45:05.745059 20919 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:33004:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:45:05.746232 20919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:45:05.754797 21195 tablet_service.cc:1467] Processing CreateTablet for tablet 010cda110c1041e3a58aec1a16a1cfcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b9c1fec0c0348ef81bb818b0b431321]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:45:05.754806 21334 tablet_service.cc:1467] Processing CreateTablet for tablet 010cda110c1041e3a58aec1a16a1cfcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b9c1fec0c0348ef81bb818b0b431321]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:45:05.755431 21334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010cda110c1041e3a58aec1a16a1cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.755424 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010cda110c1041e3a58aec1a16a1cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.756462 21055 tablet_service.cc:1467] Processing CreateTablet for tablet 010cda110c1041e3a58aec1a16a1cfcb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b9c1fec0c0348ef81bb818b0b431321]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:45:05.757005 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010cda110c1041e3a58aec1a16a1cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:05.763217 21432 tablet_bootstrap.cc:492] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: Bootstrap starting.
I20260928 15:45:05.764323 21429 tablet_bootstrap.cc:492] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: Bootstrap starting.
I20260928 15:45:05.765098 21432 tablet_bootstrap.cc:654] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.766085 21429 tablet_bootstrap.cc:654] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.776122 21428 tablet_bootstrap.cc:492] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: Bootstrap starting.
I20260928 15:45:05.776142 21432 tablet_bootstrap.cc:492] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: No bootstrap required, opened a new log
I20260928 15:45:05.776338 21432 ts_tablet_manager.cc:1397] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260928 15:45:05.776472 21429 tablet_bootstrap.cc:492] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: No bootstrap required, opened a new log
I20260928 15:45:05.776651 21429 ts_tablet_manager.cc:1397] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260928 15:45:05.777312 21432 raft_consensus.cc:348] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.777408 21429 raft_consensus.cc:348] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.777601 21432 raft_consensus.cc:374] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.777648 21429 raft_consensus.cc:374] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.777695 21432 raft_consensus.cc:729] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad6c82e1844ac9c9922653d61d922, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.777732 21429 raft_consensus.cc:729] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61fd9e7b9444c55b0966673a222b7bc, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.778029 21429 consensus_queue.cc:260] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.778126 21432 consensus_queue.cc:260] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.778555 21446 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.778566 21435 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.778959 21435 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:45:05.778959 21446 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:45:05.779486 21429 ts_tablet_manager.cc:1428] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 15:45:05.780177 21432 ts_tablet_manager.cc:1428] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 15:45:05.780583 21428 tablet_bootstrap.cc:654] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:05.785627 21428 tablet_bootstrap.cc:492] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: No bootstrap required, opened a new log
I20260928 15:45:05.785815 21428 ts_tablet_manager.cc:1397] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20260928 15:45:05.786615 21428 raft_consensus.cc:348] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.786866 21428 raft_consensus.cc:374] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:05.786948 21428 raft_consensus.cc:729] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b90f9e4e2dd48b9b914eaba542405db, State: Initialized, Role: FOLLOWER
I20260928 15:45:05.787241 21428 consensus_queue.cc:260] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.787673 21434 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.787990 21434 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:45:05.789131 21428 ts_tablet_manager.cc:1428] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 15:45:05.806330 21446 raft_consensus.cc:482] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:45:05.806536 21446 raft_consensus.cc:504] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.807384 21446 leader_election.cc:288] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741)
I20260928 15:45:05.808055 21215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010cda110c1041e3a58aec1a16a1cfcb" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b90f9e4e2dd48b9b914eaba542405db" is_pre_election: true
I20260928 15:45:05.808053 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010cda110c1041e3a58aec1a16a1cfcb" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc" is_pre_election: true
I20260928 15:45:05.808343 21080 raft_consensus.cc:2454] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 0.
I20260928 15:45:05.808331 21215 raft_consensus.cc:2454] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 0.
I20260928 15:45:05.808925 21288 leader_election.cc:302] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b90f9e4e2dd48b9b914eaba542405db, f2fad6c82e1844ac9c9922653d61d922; no voters:
I20260928 15:45:05.809244 21446 raft_consensus.cc:2781] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:45:05.809372 21446 raft_consensus.cc:482] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:45:05.809465 21446 raft_consensus.cc:3037] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.812682 21446 raft_consensus.cc:504] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.813428 21446 leader_election.cc:288] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 election: Requested vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741)
I20260928 15:45:05.814136 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010cda110c1041e3a58aec1a16a1cfcb" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc"
I20260928 15:45:05.814154 21215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010cda110c1041e3a58aec1a16a1cfcb" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b90f9e4e2dd48b9b914eaba542405db"
I20260928 15:45:05.814389 21215 raft_consensus.cc:3037] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.814416 21080 raft_consensus.cc:3037] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:05.817518 21080 raft_consensus.cc:2454] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 1.
I20260928 15:45:05.817538 21215 raft_consensus.cc:2454] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 1.
I20260928 15:45:05.818011 21287 leader_election.cc:302] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad6c82e1844ac9c9922653d61d922, f61fd9e7b9444c55b0966673a222b7bc; no voters:
I20260928 15:45:05.818342 21446 raft_consensus.cc:2781] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:05.818491 21446 raft_consensus.cc:686] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [term 1 LEADER]: Becoming Leader. State: Replica: f2fad6c82e1844ac9c9922653d61d922, State: Running, Role: LEADER
I20260928 15:45:05.818749 21446 consensus_queue.cc:237] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:05.820523 21436 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: TxnStatusTablet state changed. Reason: New leader f2fad6c82e1844ac9c9922653d61d922. Latest consensus state: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.820796 21436 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:45:05.821557 21471 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:45:05.821798 21471 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:45:05.822291 20918 catalog_manager.cc:5495] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } health_report { overall_health: HEALTHY } } }
I20260928 15:45:05.842278 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.836276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.13.5.67:39657)
0928 15:45:05.836486 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.836494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.836621 (+ 127us) server_negotiation.cc:407] Connection header received
0928 15:45:05.836932 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.836936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.837007 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.837088 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:05.837538 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.838322 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.840330 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.840871 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.840995 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.841610 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:05.841724 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:05.841812 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.841986 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.842056 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 15:45:05.842478 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.836256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39657 (local address 127.0.0.1:58854)
0928 15:45:05.836523 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.836545 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.836704 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.837156 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.837162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.837177 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:45:05.837397 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.837409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.838457 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.838463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.840138 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.840154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.841055 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.841070 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.841291 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.841394 (+ 103us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:45:05.841841 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:45:05.841865 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.841956 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.842148 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 15:45:05.846331 21215 raft_consensus.cc:1261] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Refusing update from remote peer f2fad6c82e1844ac9c9922653d61d922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:05.846347 21080 raft_consensus.cc:1261] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Refusing update from remote peer f2fad6c82e1844ac9c9922653d61d922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:05.846957 21446 consensus_queue.cc:1035] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:05.847249 21436 consensus_queue.cc:1035] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:05.853056 21435 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: TxnStatusTablet state changed. Reason: New leader f2fad6c82e1844ac9c9922653d61d922. Latest consensus state: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.853392 21435 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:45:05.852336 21434 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: TxnStatusTablet state changed. Reason: New leader f2fad6c82e1844ac9c9922653d61d922. Latest consensus state: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.854161 21434 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:45:05.855244 21436 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.856451 21436 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:45:05.856724 21441 mvcc.cc:205] Tried to move back new op lower bound from 7334339812738121728 to 7334339812633731072. Current Snapshot: MvccSnapshot[applied={T|T < 7334339812738121728 or (T in {7334339812738121728})}]
I20260928 15:45:05.857499 21438 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.857844 21438 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P f2fad6c82e1844ac9c9922653d61d922: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:45:05.858475 21443 mvcc.cc:205] Tried to move back new op lower bound from 7334339812738121728 to 7334339812633731072. Current Snapshot: MvccSnapshot[applied={T|T < 7334339812738121728}]
I20260928 15:45:05.858563 21434 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.858845 21434 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P 3b90f9e4e2dd48b9b914eaba542405db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:45:05.862267 21435 tablet_replica.cc:439] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } } }
I20260928 15:45:05.862514 21435 tablet_replica.cc:442] T 010cda110c1041e3a58aec1a16a1cfcb P f61fd9e7b9444c55b0966673a222b7bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:45:05.869369 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.863095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.13.5.66:46741)
0928 15:45:05.863250 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.863258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.863277 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:45:05.863367 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.863372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.863502 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.863682 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:05.864310 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.865476 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.866760 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.867429 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.867561 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.868142 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:05.868785 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:05.868941 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.869002 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.869082 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:45:05.885866 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.879938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:32915 (local address 127.13.5.67:39657)
0928 15:45:05.880120 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.880124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.880636 (+ 512us) server_negotiation.cc:407] Connection header received
0928 15:45:05.880855 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.880864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.880932 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.881027 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:05.881519 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.882304 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.883886 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.884447 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.884603 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.885026 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:05.885182 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:05.885286 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.885429 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.885519 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:45:05.885929 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.879937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39657 (local address 127.13.5.66:32915)
0928 15:45:05.880532 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.880548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.880711 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.881042 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.881047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.881057 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:45:05.881361 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.881369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.882443 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.882449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.883722 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.883734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.884628 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.884634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.884783 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.884833 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:45:05.885285 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:45:05.885299 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.885405 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.885563 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":340,"threads_started":1}
I20260928 15:45:05.895470 21354 raft_consensus.cc:1261] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Refusing update from remote peer 3b90f9e4e2dd48b9b914eaba542405db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:05.895526 21080 raft_consensus.cc:1261] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Refusing update from remote peer 3b90f9e4e2dd48b9b914eaba542405db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:05.896253 21434 consensus_queue.cc:1035] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:05.896585 21459 consensus_queue.cc:1035] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:05.939425 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.932226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:58235 (local address 127.13.5.66:46741)
0928 15:45:05.932410 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:05.932418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:05.933348 (+ 930us) server_negotiation.cc:407] Connection header received
0928 15:45:05.933576 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:05.933582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:05.933685 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:05.933783 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:05.934609 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.935312 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.937634 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.938058 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.938164 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.938808 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:05.938908 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:05.938972 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:05.939156 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 15:45:05.939222 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:45:05.939693 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:05.932222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:46741 (local address 127.13.5.67:58235)
0928 15:45:05.933243 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:05.933263 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:05.933424 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:05.933975 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:05.933979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:05.933992 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:45:05.934403 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.934427 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.935460 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.935467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.937406 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:05.937422 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:05.938326 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:05.938342 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:05.938539 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:05.938625 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:45:05.939020 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:45:05.939032 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:45:05.939131 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:45:05.939279 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":648,"threads_started":1}
I20260928 15:45:05.981242 20918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:05.981526 20918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:05.985309 20918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: Sending DeleteTablet for 3 replicas of tablet f5121e2011a1476baeb92e3407e36de2
I20260928 15:45:05.986243 21055 tablet_service.cc:1514] Processing DeleteTablet for tablet f5121e2011a1476baeb92e3407e36de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:05 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:54592
I20260928 15:45:05.986542 21452 tablet_replica.cc:330] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: stopping tablet replica
I20260928 15:45:05.986743 21452 raft_consensus.cc:2229] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:05.986953 21452 raft_consensus.cc:2258] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:05.987746 21195 tablet_service.cc:1514] Processing DeleteTablet for tablet f5121e2011a1476baeb92e3407e36de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:05 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52242
I20260928 15:45:05.988114 21453 tablet_replica.cc:330] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: stopping tablet replica
I20260928 15:45:05.988340 21453 raft_consensus.cc:2229] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:45:05.988924 21453 raft_consensus.cc:2258] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:05.988977 21334 tablet_service.cc:1514] Processing DeleteTablet for tablet f5121e2011a1476baeb92e3407e36de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:05 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:58828
I20260928 15:45:05.989297 21451 tablet_replica.cc:330] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: stopping tablet replica
I20260928 15:45:05.989502 21451 raft_consensus.cc:2229] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:05.989717 21451 raft_consensus.cc:2258] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:05.990285 21452 ts_tablet_manager.cc:1905] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:05.990301 21453 ts_tablet_manager.cc:1905] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:05.997259 21451 ts_tablet_manager.cc:1905] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:05.998302 21453 ts_tablet_manager.cc:1918] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:45:05.998503 21453 log.cc:1196] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/f5121e2011a1476baeb92e3407e36de2
I20260928 15:45:05.998941 21453 ts_tablet_manager.cc:1939] T f5121e2011a1476baeb92e3407e36de2 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting consensus metadata
I20260928 15:45:06.000196 20906 catalog_manager.cc:4841] TS 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741): tablet f5121e2011a1476baeb92e3407e36de2 (table test-table [id=9716a50555cb4616af1990a712a2594a]) successfully deleted
I20260928 15:45:06.005597 21451 ts_tablet_manager.cc:1918] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:45:06.005786 21451 log.cc:1196] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/f5121e2011a1476baeb92e3407e36de2
I20260928 15:45:06.005759 21452 ts_tablet_manager.cc:1918] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:45:06.005929 21452 log.cc:1196] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/f5121e2011a1476baeb92e3407e36de2
I20260928 15:45:06.006245 21451 ts_tablet_manager.cc:1939] T f5121e2011a1476baeb92e3407e36de2 P f2fad6c82e1844ac9c9922653d61d922: Deleting consensus metadata
I20260928 15:45:06.006376 21452 ts_tablet_manager.cc:1939] T f5121e2011a1476baeb92e3407e36de2 P f61fd9e7b9444c55b0966673a222b7bc: Deleting consensus metadata
I20260928 15:45:06.007725 20905 catalog_manager.cc:4841] TS f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775): tablet f5121e2011a1476baeb92e3407e36de2 (table test-table [id=9716a50555cb4616af1990a712a2594a]) successfully deleted
I20260928 15:45:06.009196 20906 catalog_manager.cc:4841] TS f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657): tablet f5121e2011a1476baeb92e3407e36de2 (table test-table [id=9716a50555cb4616af1990a712a2594a]) successfully deleted
Sep 28 15:45:06 dist-test-slave-j968 krb5kdc[20874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:45:06 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:06 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:45:06.071728 21014 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:06.058087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.13.5.126:37011)
0928 15:45:06.058291 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:06.058298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:06.058831 (+ 533us) server_negotiation.cc:407] Connection header received
0928 15:45:06.059017 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:06.059022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:06.059110 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:06.059221 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:06.061025 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.062151 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.064115 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.064811 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.064958 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.071453 (+ 6495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33016: BlockingRecv error: recv got EOF from 127.0.0.1:33016 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 15:45:06 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM
I20260928 15:45:06.099810 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:06.081376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.13.5.126:37011)
0928 15:45:06.081575 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:06.081581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:06.082099 (+ 518us) server_negotiation.cc:407] Connection header received
0928 15:45:06.082289 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:06.082297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:06.082405 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:06.082535 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:06.084191 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.085368 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.087220 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.087830 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.087986 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.093286 (+ 5300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:06.093294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:06.093299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:06.093344 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:06.096841 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:06.097662 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:06.097667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:06.097671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:06.097846 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:06.098267 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:06.098292 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:06.098295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:06.098634 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:06.098880 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:06.099196 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 15:45:06.099422 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:45:06.104048 20918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:45:06.104789 20918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:45:06.113082 21055 tablet_service.cc:1467] Processing CreateTablet for tablet 1e92f47b39d84e00bfdf7245e833657c (DEFAULT_TABLE table=test-table [id=12f095c9bc2c443999ae014410867c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:06.113225 21195 tablet_service.cc:1467] Processing CreateTablet for tablet 1e92f47b39d84e00bfdf7245e833657c (DEFAULT_TABLE table=test-table [id=12f095c9bc2c443999ae014410867c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:06.113631 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e92f47b39d84e00bfdf7245e833657c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:06.113654 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e92f47b39d84e00bfdf7245e833657c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:06.115675 21334 tablet_service.cc:1467] Processing CreateTablet for tablet 1e92f47b39d84e00bfdf7245e833657c (DEFAULT_TABLE table=test-table [id=12f095c9bc2c443999ae014410867c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:06.116163 21334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e92f47b39d84e00bfdf7245e833657c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:06.124795 21428 tablet_bootstrap.cc:492] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: Bootstrap starting.
I20260928 15:45:06.126698 21428 tablet_bootstrap.cc:654] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:06.129562 21429 tablet_bootstrap.cc:492] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: Bootstrap starting.
I20260928 15:45:06.131297 21429 tablet_bootstrap.cc:654] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:06.137154 21432 tablet_bootstrap.cc:492] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: Bootstrap starting.
I20260928 15:45:06.138979 21432 tablet_bootstrap.cc:654] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:06.145627 21432 tablet_bootstrap.cc:492] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: No bootstrap required, opened a new log
I20260928 15:45:06.145799 21432 ts_tablet_manager.cc:1397] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20260928 15:45:06.146651 21432 raft_consensus.cc:348] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.146939 21432 raft_consensus.cc:374] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:06.147032 21432 raft_consensus.cc:729] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad6c82e1844ac9c9922653d61d922, State: Initialized, Role: FOLLOWER
I20260928 15:45:06.148002 21432 consensus_queue.cc:260] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.148809 21432 ts_tablet_manager.cc:1428] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 15:45:06.154009 21428 tablet_bootstrap.cc:492] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: No bootstrap required, opened a new log
I20260928 15:45:06.154194 21428 ts_tablet_manager.cc:1397] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.001s
I20260928 15:45:06.154268 21429 tablet_bootstrap.cc:492] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: No bootstrap required, opened a new log
I20260928 15:45:06.154431 21429 ts_tablet_manager.cc:1397] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.007s
I20260928 15:45:06.155072 21428 raft_consensus.cc:348] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.155218 21429 raft_consensus.cc:348] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.155407 21428 raft_consensus.cc:374] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:06.155484 21429 raft_consensus.cc:374] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:06.155503 21428 raft_consensus.cc:729] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b90f9e4e2dd48b9b914eaba542405db, State: Initialized, Role: FOLLOWER
I20260928 15:45:06.155586 21429 raft_consensus.cc:729] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61fd9e7b9444c55b0966673a222b7bc, State: Initialized, Role: FOLLOWER
I20260928 15:45:06.155781 21428 consensus_queue.cc:260] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.155851 21429 consensus_queue.cc:260] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.156546 21429 ts_tablet_manager.cc:1428] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 15:45:06.160408 21428 ts_tablet_manager.cc:1428] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260928 15:45:06.221222 21459 raft_consensus.cc:482] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:45:06.221534 21459 raft_consensus.cc:504] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.222334 21459 leader_election.cc:288] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657)
I20260928 15:45:06.223165 21354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e92f47b39d84e00bfdf7245e833657c" candidate_uuid: "3b90f9e4e2dd48b9b914eaba542405db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad6c82e1844ac9c9922653d61d922" is_pre_election: true
I20260928 15:45:06.223471 21354 raft_consensus.cc:2454] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b90f9e4e2dd48b9b914eaba542405db in term 0.
I20260928 15:45:06.224068 21149 leader_election.cc:302] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b90f9e4e2dd48b9b914eaba542405db, f2fad6c82e1844ac9c9922653d61d922; no voters:
I20260928 15:45:06.224907 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e92f47b39d84e00bfdf7245e833657c" candidate_uuid: "3b90f9e4e2dd48b9b914eaba542405db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc" is_pre_election: true
I20260928 15:45:06.225178 21080 raft_consensus.cc:2454] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b90f9e4e2dd48b9b914eaba542405db in term 0.
I20260928 15:45:06.225854 21459 raft_consensus.cc:2781] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:45:06.225983 21459 raft_consensus.cc:482] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:45:06.226106 21459 raft_consensus.cc:3037] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:06.229413 21459 raft_consensus.cc:504] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.230680 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e92f47b39d84e00bfdf7245e833657c" candidate_uuid: "3b90f9e4e2dd48b9b914eaba542405db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc"
I20260928 15:45:06.230923 21080 raft_consensus.cc:3037] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:06.234103 21080 raft_consensus.cc:2454] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b90f9e4e2dd48b9b914eaba542405db in term 1.
I20260928 15:45:06.234588 21148 leader_election.cc:302] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b90f9e4e2dd48b9b914eaba542405db, f61fd9e7b9444c55b0966673a222b7bc; no voters:
I20260928 15:45:06.234944 21484 raft_consensus.cc:2781] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:06.235123 21484 raft_consensus.cc:686] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 1 LEADER]: Becoming Leader. State: Replica: 3b90f9e4e2dd48b9b914eaba542405db, State: Running, Role: LEADER
I20260928 15:45:06.235479 21484 consensus_queue.cc:237] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.239810 21354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e92f47b39d84e00bfdf7245e833657c" candidate_uuid: "3b90f9e4e2dd48b9b914eaba542405db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad6c82e1844ac9c9922653d61d922"
I20260928 15:45:06.240090 21354 raft_consensus.cc:3037] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:06.243373 21354 raft_consensus.cc:2454] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b90f9e4e2dd48b9b914eaba542405db in term 1.
I20260928 15:45:06.244164 21459 leader_election.cc:288] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [CANDIDATE]: Term 1 election: Requested vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657)
I20260928 15:45:06.253859 20918 catalog_manager.cc:5495] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66). New cstate: current_term: 1 leader_uuid: "3b90f9e4e2dd48b9b914eaba542405db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } health_report { overall_health: UNKNOWN } } }
I20260928 15:45:06.275144 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:06.267682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.13.5.66:46741)
0928 15:45:06.268019 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:06.268025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:06.268438 (+ 413us) server_negotiation.cc:407] Connection header received
0928 15:45:06.268518 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:06.268524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:06.268646 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:06.268748 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:06.269392 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.270584 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.272696 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.273444 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.273580 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.274214 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:06.274373 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:06.274459 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:06.274712 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:45:06.274812 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260928 15:45:06.279076 21080 raft_consensus.cc:1261] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Refusing update from remote peer 3b90f9e4e2dd48b9b914eaba542405db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:06.279760 21459 consensus_queue.cc:1035] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:06.279027 21354 raft_consensus.cc:1261] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Refusing update from remote peer 3b90f9e4e2dd48b9b914eaba542405db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:06.281483 21459 consensus_queue.cc:1035] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:06.304008 20918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:06.304347 20918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:06.308138 20918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: Sending DeleteTablet for 3 replicas of tablet 1e92f47b39d84e00bfdf7245e833657c
I20260928 15:45:06.308849 21055 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e92f47b39d84e00bfdf7245e833657c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:06 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:54592
I20260928 15:45:06.309330 21452 tablet_replica.cc:330] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: stopping tablet replica
I20260928 15:45:06.309510 21452 raft_consensus.cc:2229] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:06.309693 21452 raft_consensus.cc:2258] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:06.310668 21334 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e92f47b39d84e00bfdf7245e833657c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:06 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:58828
I20260928 15:45:06.310992 21451 tablet_replica.cc:330] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: stopping tablet replica
I20260928 15:45:06.311050 21195 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e92f47b39d84e00bfdf7245e833657c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:06 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52242
I20260928 15:45:06.311178 21451 raft_consensus.cc:2229] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:06.311347 21453 tablet_replica.cc:330] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: stopping tablet replica
I20260928 15:45:06.311424 21451 raft_consensus.cc:2258] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:06.311518 21453 raft_consensus.cc:2229] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:45:06.311848 21453 raft_consensus.cc:2258] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:06.316385 21452 ts_tablet_manager.cc:1905] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:06.320256 21453 ts_tablet_manager.cc:1905] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:06.324666 21451 ts_tablet_manager.cc:1905] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:06.330621 21452 ts_tablet_manager.cc:1918] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:06.330914 21452 log.cc:1196] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/1e92f47b39d84e00bfdf7245e833657c
I20260928 15:45:06.330789 20916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:45:06.331561 20916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:45:06.332056 21451 ts_tablet_manager.cc:1918] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:06.332211 21451 log.cc:1196] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/1e92f47b39d84e00bfdf7245e833657c
I20260928 15:45:06.332651 21451 ts_tablet_manager.cc:1939] T 1e92f47b39d84e00bfdf7245e833657c P f2fad6c82e1844ac9c9922653d61d922: Deleting consensus metadata
I20260928 15:45:06.333871 20906 catalog_manager.cc:4841] TS f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657): tablet 1e92f47b39d84e00bfdf7245e833657c (table test-table [id=12f095c9bc2c443999ae014410867c5c]) successfully deleted
I20260928 15:45:06.334153 21453 ts_tablet_manager.cc:1918] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:06.334370 21453 log.cc:1196] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/1e92f47b39d84e00bfdf7245e833657c
I20260928 15:45:06.334884 21453 ts_tablet_manager.cc:1939] T 1e92f47b39d84e00bfdf7245e833657c P 3b90f9e4e2dd48b9b914eaba542405db: Deleting consensus metadata
I20260928 15:45:06.336050 20906 catalog_manager.cc:4841] TS 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741): tablet 1e92f47b39d84e00bfdf7245e833657c (table test-table [id=12f095c9bc2c443999ae014410867c5c]) successfully deleted
I20260928 15:45:06.339638 21055 tablet_service.cc:1467] Processing CreateTablet for tablet 817d720b1ccb4221a2962a7ae7a548a5 (DEFAULT_TABLE table=test-table [id=a5044e420c6f4882ab49b04da2e7ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:06.340102 21055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817d720b1ccb4221a2962a7ae7a548a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:06.343906 21452 ts_tablet_manager.cc:1939] T 1e92f47b39d84e00bfdf7245e833657c P f61fd9e7b9444c55b0966673a222b7bc: Deleting consensus metadata
I20260928 15:45:06.345060 20905 catalog_manager.cc:4841] TS f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775): tablet 1e92f47b39d84e00bfdf7245e833657c (table test-table [id=12f095c9bc2c443999ae014410867c5c]) successfully deleted
I20260928 15:45:06.345266 21195 tablet_service.cc:1467] Processing CreateTablet for tablet 817d720b1ccb4221a2962a7ae7a548a5 (DEFAULT_TABLE table=test-table [id=a5044e420c6f4882ab49b04da2e7ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:06.345369 21334 tablet_service.cc:1467] Processing CreateTablet for tablet 817d720b1ccb4221a2962a7ae7a548a5 (DEFAULT_TABLE table=test-table [id=a5044e420c6f4882ab49b04da2e7ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:06.345703 21195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817d720b1ccb4221a2962a7ae7a548a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:06.345818 21334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817d720b1ccb4221a2962a7ae7a548a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:06.352627 21428 tablet_bootstrap.cc:492] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: Bootstrap starting.
I20260928 15:45:06.354480 21428 tablet_bootstrap.cc:654] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:06.356370 21429 tablet_bootstrap.cc:492] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: Bootstrap starting.
I20260928 15:45:06.358076 21429 tablet_bootstrap.cc:654] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:06.364782 21432 tablet_bootstrap.cc:492] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: Bootstrap starting.
I20260928 15:45:06.366673 21432 tablet_bootstrap.cc:654] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:06.370983 21428 tablet_bootstrap.cc:492] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: No bootstrap required, opened a new log
I20260928 15:45:06.371170 21428 ts_tablet_manager.cc:1397] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.005s
I20260928 15:45:06.372135 21428 raft_consensus.cc:348] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.372416 21428 raft_consensus.cc:374] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:06.372505 21428 raft_consensus.cc:729] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b90f9e4e2dd48b9b914eaba542405db, State: Initialized, Role: FOLLOWER
I20260928 15:45:06.372843 21428 consensus_queue.cc:260] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.374130 21428 ts_tablet_manager.cc:1428] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:06.392778 21432 tablet_bootstrap.cc:492] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: No bootstrap required, opened a new log
I20260928 15:45:06.393007 21432 ts_tablet_manager.cc:1397] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.004s
I20260928 15:45:06.393428 21429 tablet_bootstrap.cc:492] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: No bootstrap required, opened a new log
I20260928 15:45:06.393678 21429 ts_tablet_manager.cc:1397] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.011s
I20260928 15:45:06.393899 21432 raft_consensus.cc:348] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.394183 21432 raft_consensus.cc:374] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:06.394279 21432 raft_consensus.cc:729] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad6c82e1844ac9c9922653d61d922, State: Initialized, Role: FOLLOWER
I20260928 15:45:06.394541 21432 consensus_queue.cc:260] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.395105 21432 ts_tablet_manager.cc:1428] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 15:45:06.395568 21429 raft_consensus.cc:348] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.395840 21429 raft_consensus.cc:374] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:06.395959 21429 raft_consensus.cc:729] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61fd9e7b9444c55b0966673a222b7bc, State: Initialized, Role: FOLLOWER
I20260928 15:45:06.396243 21429 consensus_queue.cc:260] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.396848 21429 ts_tablet_manager.cc:1428] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 15:45:06.454134 21446 raft_consensus.cc:482] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:45:06.454398 21446 raft_consensus.cc:504] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.455291 21446 leader_election.cc:288] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741)
I20260928 15:45:06.456027 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817d720b1ccb4221a2962a7ae7a548a5" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc" is_pre_election: true
I20260928 15:45:06.456079 21215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817d720b1ccb4221a2962a7ae7a548a5" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b90f9e4e2dd48b9b914eaba542405db" is_pre_election: true
I20260928 15:45:06.456322 21080 raft_consensus.cc:2454] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 0.
I20260928 15:45:06.456377 21215 raft_consensus.cc:2454] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 0.
I20260928 15:45:06.456908 21287 leader_election.cc:302] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad6c82e1844ac9c9922653d61d922, f61fd9e7b9444c55b0966673a222b7bc; no voters:
I20260928 15:45:06.457346 21446 raft_consensus.cc:2781] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:45:06.457612 21446 raft_consensus.cc:482] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:45:06.457803 21446 raft_consensus.cc:3037] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:06.462235 21446 raft_consensus.cc:504] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.463800 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817d720b1ccb4221a2962a7ae7a548a5" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61fd9e7b9444c55b0966673a222b7bc"
I20260928 15:45:06.464139 21080 raft_consensus.cc:3037] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:06.465282 21446 leader_election.cc:288] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 election: Requested vote from peers f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775), 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741)
I20260928 15:45:06.465734 21215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817d720b1ccb4221a2962a7ae7a548a5" candidate_uuid: "f2fad6c82e1844ac9c9922653d61d922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b90f9e4e2dd48b9b914eaba542405db"
I20260928 15:45:06.466040 21215 raft_consensus.cc:3037] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:06.468729 21080 raft_consensus.cc:2454] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 1.
I20260928 15:45:06.469249 21287 leader_election.cc:302] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad6c82e1844ac9c9922653d61d922, f61fd9e7b9444c55b0966673a222b7bc; no voters:
I20260928 15:45:06.469717 21446 raft_consensus.cc:2781] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:06.470269 21446 raft_consensus.cc:686] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 1 LEADER]: Becoming Leader. State: Replica: f2fad6c82e1844ac9c9922653d61d922, State: Running, Role: LEADER
I20260928 15:45:06.470300 21215 raft_consensus.cc:2454] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad6c82e1844ac9c9922653d61d922 in term 1.
I20260928 15:45:06.470623 21446 consensus_queue.cc:237] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } }
I20260928 15:45:06.474498 20919 catalog_manager.cc:5495] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "f2fad6c82e1844ac9c9922653d61d922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2fad6c82e1844ac9c9922653d61d922" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39657 } health_report { overall_health: HEALTHY } } }
I20260928 15:45:06.509948 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:06.502016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.13.5.67:39657)
0928 15:45:06.503133 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:06.503140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:06.503159 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:45:06.503228 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:06.503233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:06.503316 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:06.503441 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:06.503998 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.505229 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.507319 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.508179 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.508333 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.508970 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:06.509136 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:06.509243 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:06.509452 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 15:45:06.509559 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":811,"threads_started":1}
I20260928 15:45:06.547559 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:06.531645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:60625 (local address 127.13.5.67:39657)
0928 15:45:06.531885 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:06.531895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:06.540835 (+ 8940us) server_negotiation.cc:407] Connection header received
0928 15:45:06.541050 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:06.541056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:06.541142 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:06.541232 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:06.541953 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.543061 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.545302 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.546030 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.546124 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.546753 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:06.546911 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:06.546974 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:06.547158 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 15:45:06.547247 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:45:06.548602 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:06.531285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39657 (local address 127.13.5.67:60625)
0928 15:45:06.540690 (+ 9405us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:06.540718 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:06.540892 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:06.541410 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:06.541415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:06.541428 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:45:06.541783 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:06.541797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.543242 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.543247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:06.545135 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:06.545151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.546330 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.546337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:06.546528 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.546581 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:45:06.547037 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:45:06.547047 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:45:06.548108 (+ 1061us) client_negotiation.cc:240] Negotiation successful
0928 15:45:06.548242 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9195,"thread_start_us":9022,"threads_started":1}
I20260928 15:45:06.559492 21080 raft_consensus.cc:1261] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Refusing update from remote peer f2fad6c82e1844ac9c9922653d61d922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:06.559590 21215 raft_consensus.cc:1261] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Refusing update from remote peer f2fad6c82e1844ac9c9922653d61d922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:06.560225 21475 consensus_queue.cc:1035] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b90f9e4e2dd48b9b914eaba542405db" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 46741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:06.560750 21475 consensus_queue.cc:1035] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61fd9e7b9444c55b0966673a222b7bc" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:06.597848 21442 mvcc.cc:205] Tried to move back new op lower bound from 7334339815661658112 to 7334339815307034624. Current Snapshot: MvccSnapshot[applied={T|T < 7334339815661658112}]
I20260928 15:45:06.635303 20919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:06.635594 20919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:06.638423 20919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a64bdeb07faf47ce9b957c53e1c5d4ae: Sending DeleteTablet for 3 replicas of tablet 817d720b1ccb4221a2962a7ae7a548a5
I20260928 15:45:06.639256 21195 tablet_service.cc:1514] Processing DeleteTablet for tablet 817d720b1ccb4221a2962a7ae7a548a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:06 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52242
I20260928 15:45:06.639235 21055 tablet_service.cc:1514] Processing DeleteTablet for tablet 817d720b1ccb4221a2962a7ae7a548a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:06 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:54592
I20260928 15:45:06.639535 21452 tablet_replica.cc:330] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: stopping tablet replica
I20260928 15:45:06.639546 21453 tablet_replica.cc:330] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: stopping tablet replica
I20260928 15:45:06.639739 21453 raft_consensus.cc:2229] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:06.639739 21452 raft_consensus.cc:2229] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:06.639725 21333 tablet_service.cc:1514] Processing DeleteTablet for tablet 817d720b1ccb4221a2962a7ae7a548a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:06 UTC) from {username='oryx', principal='oryx/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:58828
I20260928 15:45:06.639959 21453 raft_consensus.cc:2258] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:06.639968 21452 raft_consensus.cc:2258] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:06.640043 21451 tablet_replica.cc:330] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: stopping tablet replica
I20260928 15:45:06.640245 21451 raft_consensus.cc:2229] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:45:06.640549 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 20990
I20260928 15:45:06.640587 21451 raft_consensus.cc:2258] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:06.641086 21452 ts_tablet_manager.cc:1905] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:06.641119 21453 ts_tablet_manager.cc:1905] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:06.645705 21451 ts_tablet_manager.cc:1905] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:06.647847 21452 ts_tablet_manager.cc:1918] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:45:06.648046 21452 log.cc:1196] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/817d720b1ccb4221a2962a7ae7a548a5
I20260928 15:45:06.648386 21453 ts_tablet_manager.cc:1918] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:45:06.648450 21452 ts_tablet_manager.cc:1939] T 817d720b1ccb4221a2962a7ae7a548a5 P f61fd9e7b9444c55b0966673a222b7bc: Deleting consensus metadata
I20260928 15:45:06.648545 21453 log.cc:1196] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/817d720b1ccb4221a2962a7ae7a548a5
I20260928 15:45:06.648952 21453 ts_tablet_manager.cc:1939] T 817d720b1ccb4221a2962a7ae7a548a5 P 3b90f9e4e2dd48b9b914eaba542405db: Deleting consensus metadata
I20260928 15:45:06.649552 20905 catalog_manager.cc:4841] TS f61fd9e7b9444c55b0966673a222b7bc (127.13.5.65:42775): tablet 817d720b1ccb4221a2962a7ae7a548a5 (table test-table [id=a5044e420c6f4882ab49b04da2e7ba7c]) successfully deleted
I20260928 15:45:06.650278 20906 catalog_manager.cc:4841] TS 3b90f9e4e2dd48b9b914eaba542405db (127.13.5.66:46741): tablet 817d720b1ccb4221a2962a7ae7a548a5 (table test-table [id=a5044e420c6f4882ab49b04da2e7ba7c]) successfully deleted
I20260928 15:45:06.651839 21451 ts_tablet_manager.cc:1918] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:45:06.652050 21451 log.cc:1196] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/817d720b1ccb4221a2962a7ae7a548a5
I20260928 15:45:06.652424 21451 ts_tablet_manager.cc:1939] T 817d720b1ccb4221a2962a7ae7a548a5 P f2fad6c82e1844ac9c9922653d61d922: Deleting consensus metadata
I20260928 15:45:06.653296 20906 catalog_manager.cc:4841] TS f2fad6c82e1844ac9c9922653d61d922 (127.13.5.67:39657): tablet 817d720b1ccb4221a2962a7ae7a548a5 (table test-table [id=a5044e420c6f4882ab49b04da2e7ba7c]) successfully deleted
Sep 28 15:45:06 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.65@KRBTEST.COM
I20260928 15:45:06.660403 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:06.641870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.13.5.65:42775)
0928 15:45:06.644020 (+ 2150us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:06.644025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:06.644040 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:45:06.644083 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:06.644086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:06.644142 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:06.644257 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:06.646381 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.647469 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.649461 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.650133 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.650303 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.654605 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:06.654620 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:06.654638 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:06.654678 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:06.657759 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:06.658440 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:06.658449 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:06.658453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:06.658595 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:06.659008 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:06.659012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:06.659014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:06.659358 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:06.659616 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:06.659969 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 15:45:06.660142 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2033,"thread_start_us":1811,"threads_started":1}
I20260928 15:45:06.661108 21121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:06.762832 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 20990
I20260928 15:45:06.791810 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 21130
Sep 28 15:45:06 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.66@KRBTEST.COM
I20260928 15:45:06.818327 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:06.792648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.13.5.66:46741)
0928 15:45:06.793165 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:06.793169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:06.793186 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:45:06.793240 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:06.793244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:06.793301 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:06.793401 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:06.795113 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.796022 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.798119 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.798547 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.798652 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.809081 (+ 10429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:06.809109 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:06.809141 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:06.809211 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:06.814090 (+ 4879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:06.815330 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:06.815346 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:06.815352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:06.815567 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:06.816138 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:06.816145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:06.816151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:06.816717 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:06.817105 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:06.817485 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 15:45:06.817783 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":313,"threads_started":1}
I20260928 15:45:06.819478 21260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:06.964489 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 21130
I20260928 15:45:06.988770 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 21269
Sep 28 15:45:06 dist-test-slave-j968 krb5kdc[20874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.5.67@KRBTEST.COM
I20260928 15:45:07.004441 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:06.989665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.13.5.67:39657)
0928 15:45:06.989861 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:06.989865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:06.989910 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:45:06.990124 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:06.990127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:06.990195 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:06.990278 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:06.991493 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.992254 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.993464 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:06.993857 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:06.993957 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:06.998370 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:06.998384 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:06.998404 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:06.998436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:07.001613 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:07.002289 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:07.002302 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:07.002304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:07.002457 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:07.002835 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:07.002841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:07.002844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:07.003213 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:07.003533 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:07.003809 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 15:45:07.004060 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:45:07.005079 21399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:07.160398 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:07.159340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42775 (local address 127.13.5.67:51225)
0928 15:45:07.160074 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:07.160202 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":514,"threads_started":1}
W20260928 15:45:07.160686 21287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42775: connect: Connection refused (error 111)
I20260928 15:45:07.161316 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 21269
I20260928 15:45:07.183923 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 20887
I20260928 15:45:07.185184 20959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:07.304943 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 20887
2026-09-28T15:45:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5631 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:45:07 dist-test-slave-j968 krb5kdc[21529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:45:07 dist-test-slave-j968 krb5kdc[21529](info): set up 2 sockets
Sep 28 15:45:07 dist-test-slave-j968 krb5kdc[21529](info): commencing operation
krb5kdc: starting...
W20260928 15:45:08.507377 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:45:08 dist-test-slave-j968 krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:45:08.736348 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36645
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:36645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:09.068234 21542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:09.068536 21542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:09.068719 21542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:09.076097 21542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:45:09.076212 21542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:09.076319 21542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:09.076403 21542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:45:09.076453 21542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:45:09.086416 21542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:36645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:09.087775 21542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:09.089181 21542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:09.101186 21547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:09.101240 21548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:09.101263 21550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:09.102100 21542 server_base.cc:1034] running on GCE node
I20260928 15:45:09.102926 21542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:09.104581 21542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:09.105857 21542 hybrid_clock.cc:648] HybridClock initialized: now 1790610309105788 us; error 61 us; skew 500 ppm
I20260928 15:45:09.106335 21542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T15:45:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1876 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:45:09 dist-test-slave-j968 krb5kdc[21556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:45:09 dist-test-slave-j968 krb5kdc[21556](info): set up 2 sockets
Sep 28 15:45:09 dist-test-slave-j968 krb5kdc[21556](info): commencing operation
krb5kdc: starting...
W20260928 15:45:10.287639 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:45:10 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:45:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:45:10.520090 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43743
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:10.861153 21569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:10.861570 21569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:10.861733 21569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:10.869381 21569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:45:10.869558 21569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:10.869673 21569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:10.869781 21569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:45:10.869856 21569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:45:10.880028 21569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:10.881516 21569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:10.882884 21569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:10.895721 21575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:10.895849 21574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:10.895699 21577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:10.897166 21569 server_base.cc:1034] running on GCE node
I20260928 15:45:10.898247 21569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:10.900166 21569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:10.901474 21569 hybrid_clock.cc:648] HybridClock initialized: now 1790610310901386 us; error 79 us; skew 500 ppm
I20260928 15:45:10.901871 21569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:10 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610310, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:10.905314 21569 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:45:10.906929 21569 webserver.cc:460] Webserver started at http://127.13.5.126:40523/ using document root <none> and password file <none>
I20260928 15:45:10.907666 21569 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:10.907814 21569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:10.908174 21569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:10.911280 21569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "f6c31b41ed014d8a8e3f4d50f37073c1"
format_stamp: "Formatted at 2026-09-28 15:45:10 on dist-test-slave-j968"
server_key: "a8492c07b74e5939c2fb854064cf8942"
server_key_iv: "2ef3830eb30969a086d64d64d4111e7c"
server_key_version: "encryptionkey@0"
I20260928 15:45:10.912130 21569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "f6c31b41ed014d8a8e3f4d50f37073c1"
format_stamp: "Formatted at 2026-09-28 15:45:10 on dist-test-slave-j968"
server_key: "a8492c07b74e5939c2fb854064cf8942"
server_key_iv: "2ef3830eb30969a086d64d64d4111e7c"
server_key_version: "encryptionkey@0"
I20260928 15:45:10.917641 21569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:45:10.921459 21584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:10.923242 21569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:45:10.923462 21569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "f6c31b41ed014d8a8e3f4d50f37073c1"
format_stamp: "Formatted at 2026-09-28 15:45:10 on dist-test-slave-j968"
server_key: "a8492c07b74e5939c2fb854064cf8942"
server_key_iv: "2ef3830eb30969a086d64d64d4111e7c"
server_key_version: "encryptionkey@0"
I20260928 15:45:10.923732 21569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:10.942194 21569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:10.946600 21569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:10.946907 21569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:10.966058 21569 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43743
I20260928 15:45:10.966091 21635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43743 every 8 connection(s)
I20260928 15:45:10.967597 21569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:45:10.971410 21636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:10.973203 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 21569
I20260928 15:45:10.973459 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:45:10.974119 13333 external_mini_cluster.cc:1428] Setting key 8263062d9d647313e8d1af6a4ee5a368
I20260928 15:45:10.983937 21636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Bootstrap starting.
I20260928 15:45:10.987975 21636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:10.989235 21636 log.cc:824] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:45:10 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:45:10.992899 21636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: No bootstrap required, opened a new log
I20260928 15:45:10.998799 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:10.976160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.13.5.126:43743)
0928 15:45:10.976971 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:10.976983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:10.977028 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:45:10.978714 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:10.978736 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:10.979129 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:10.979509 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:10.981452 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:10.982871 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:10.984767 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:10.985477 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:10.985650 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:10.990863 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:10.990877 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:10.990901 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:10.990955 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:10.994323 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:10.995266 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:10.995273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:10.995279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:10.995457 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:10.995980 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:10.995984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:10.995986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:10.996592 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:10.996827 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:10.997269 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 15:45:10.997679 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":370,"threads_started":1}
I20260928 15:45:11.002533 21636 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } }
I20260928 15:45:11.002929 21636 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:11.003041 21636 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c31b41ed014d8a8e3f4d50f37073c1, State: Initialized, Role: FOLLOWER
I20260928 15:45:11.003649 21636 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } }
I20260928 15:45:11.003943 21636 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:45:11.004091 21636 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:45:11.004268 21636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:11.006697 21636 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } }
I20260928 15:45:11.007213 21636 leader_election.cc:302] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6c31b41ed014d8a8e3f4d50f37073c1; no voters:
I20260928 15:45:11.007834 21636 leader_election.cc:288] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:45:11.008083 21641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:11.008860 21641 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 1 LEADER]: Becoming Leader. State: Replica: f6c31b41ed014d8a8e3f4d50f37073c1, State: Running, Role: LEADER
I20260928 15:45:11.009423 21641 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } }
I20260928 15:45:11.010021 21636 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:45:11.012075 21643 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6c31b41ed014d8a8e3f4d50f37073c1. Latest consensus state: current_term: 1 leader_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } } }
I20260928 15:45:11.012529 21643 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:11.016897 21642 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } } }
I20260928 15:45:11.017235 21642 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: This master's current role is: LEADER
W20260928 15:45:11.021806 21657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:45:11.021942 21657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:45:11.022114 21647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:45:11.023958 21647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:45:11.032392 21647 catalog_manager.cc:1347] Generated new cluster ID: 8e9df99689794b6caf87ce0f39e30552
I20260928 15:45:11.032513 21647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:45:11.057672 21647 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:45:11.060047 21647 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:45:11.071925 21647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Generated new TSK 0
I20260928 15:45:11.073129 21647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:45:11.144583 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43743
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:11.509231 21663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:11.509541 21663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:11.509668 21663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:11.517043 21663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:11.517150 21663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:11.517263 21663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:45:11.527117 21663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:11.528720 21663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:11.530129 21663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:11.543848 21669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:11.543816 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:11.543931 21668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:11.544705 21663 server_base.cc:1034] running on GCE node
I20260928 15:45:11.545308 21663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:11.546245 21663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:11.547528 21663 hybrid_clock.cc:648] HybridClock initialized: now 1790610311547457 us; error 61 us; skew 500 ppm
I20260928 15:45:11.547847 21663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:11 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610311, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:11.551156 21663 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:45:11.552773 21663 webserver.cc:460] Webserver started at http://127.13.5.65:34041/ using document root <none> and password file <none>
I20260928 15:45:11.553479 21663 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:11.553627 21663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:11.553933 21663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:11.556957 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "ceaf29b3123c4c448d9afda233e12f1d"
format_stamp: "Formatted at 2026-09-28 15:45:11 on dist-test-slave-j968"
server_key: "24fd8e594c751529b4f440dd4ffd1697"
server_key_iv: "b26516ba3d823d08b149f9acdd12ba34"
server_key_version: "encryptionkey@0"
I20260928 15:45:11.557749 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "ceaf29b3123c4c448d9afda233e12f1d"
format_stamp: "Formatted at 2026-09-28 15:45:11 on dist-test-slave-j968"
server_key: "24fd8e594c751529b4f440dd4ffd1697"
server_key_iv: "b26516ba3d823d08b149f9acdd12ba34"
server_key_version: "encryptionkey@0"
I20260928 15:45:11.563244 21663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:45:11.567078 21678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:11.568683 21663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:45:11.568882 21663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "ceaf29b3123c4c448d9afda233e12f1d"
format_stamp: "Formatted at 2026-09-28 15:45:11 on dist-test-slave-j968"
server_key: "24fd8e594c751529b4f440dd4ffd1697"
server_key_iv: "b26516ba3d823d08b149f9acdd12ba34"
server_key_version: "encryptionkey@0"
I20260928 15:45:11.569116 21663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:11.593286 21663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:11.598157 21663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:11.598486 21663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:11.599575 21663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:11.601037 21663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:11.601145 21663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:11.601295 21663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:11.601384 21663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:11.636526 21663 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:42141
I20260928 15:45:11.636616 21790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:42141 every 8 connection(s)
I20260928 15:45:11.637898 21663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:45:11.641722 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 21663
I20260928 15:45:11.641928 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:45:11.642680 13333 external_mini_cluster.cc:1428] Setting key 0ed7a473665f3f039ede6af765d73cbd
Sep 28 15:45:11 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610311, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:45:11.659137 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:11.640643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:52437 (local address 127.13.5.126:43743)
0928 15:45:11.641169 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:11.641174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:11.642101 (+ 927us) server_negotiation.cc:407] Connection header received
0928 15:45:11.644463 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:11.644467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:11.644535 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:11.644679 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:11.646412 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:11.647225 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:11.649030 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:11.649474 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:11.649627 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:11.653229 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:11.653234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:11.653238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:11.653271 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:11.656013 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:11.656759 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:11.656765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:11.656768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:11.656916 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:11.657352 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:11.657358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:11.657361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:11.657645 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:11.657809 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:11.658521 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 15:45:11.658751 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":265,"threads_started":1}
I20260928 15:45:11.660390 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:11.641019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:52437)
0928 15:45:11.641875 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:11.641929 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:11.644155 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:11.644993 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:11.645007 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:11.645601 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:11.646229 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:11.646248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:11.647401 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:11.647406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:11.648851 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:11.648864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:11.649721 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:11.649731 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:11.649983 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:11.650743 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:11.650766 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:11.652972 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:11.656156 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:11.656165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:11.656183 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:11.656628 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:11.657030 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:11.657036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:11.657039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:11.657232 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:11.657786 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:11.657794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:11.658138 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 15:45:11.658404 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 15:45:11.658969 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":345,"threads_started":1}
I20260928 15:45:11.662252 21791 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43743
I20260928 15:45:11.662693 21791 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:11.663450 21791 heartbeater.cc:511] Master 127.13.5.126:43743 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:45:11.665992 21601 ts_manager.cc:194] Registered new tserver with Master: ceaf29b3123c4c448d9afda233e12f1d (127.13.5.65:42141)
I20260928 15:45:11.667539 21601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:52437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:45:11.700028 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43743
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:12.045715 21797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:12.046059 21797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:12.046238 21797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:12.054191 21797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:12.054675 21797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:12.054878 21797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:45:12.075021 21797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:12.076977 21797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:12.078889 21797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:12.098237 21803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:12.101226 21802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:12.101536 21805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:12.102820 21797 server_base.cc:1034] running on GCE node
I20260928 15:45:12.103718 21797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:12.104864 21797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:12.106088 21797 hybrid_clock.cc:648] HybridClock initialized: now 1790610312106040 us; error 30 us; skew 500 ppm
I20260928 15:45:12.106506 21797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:12 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610312, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:12.110054 21797 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:45:12.111840 21797 webserver.cc:460] Webserver started at http://127.13.5.66:45623/ using document root <none> and password file <none>
I20260928 15:45:12.112650 21797 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:12.112779 21797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:12.113111 21797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:12.116220 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "5a9dad2cc18b4a29aac6e59b79ba62ad"
format_stamp: "Formatted at 2026-09-28 15:45:12 on dist-test-slave-j968"
server_key: "70942bd8609f2e36d3ae4369ab933717"
server_key_iv: "bc97946c62ea7e6f84bab4cb32320b4f"
server_key_version: "encryptionkey@0"
I20260928 15:45:12.117041 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "5a9dad2cc18b4a29aac6e59b79ba62ad"
format_stamp: "Formatted at 2026-09-28 15:45:12 on dist-test-slave-j968"
server_key: "70942bd8609f2e36d3ae4369ab933717"
server_key_iv: "bc97946c62ea7e6f84bab4cb32320b4f"
server_key_version: "encryptionkey@0"
I20260928 15:45:12.122344 21797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 15:45:12.126142 21812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:12.127692 21797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:45:12.127904 21797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "5a9dad2cc18b4a29aac6e59b79ba62ad"
format_stamp: "Formatted at 2026-09-28 15:45:12 on dist-test-slave-j968"
server_key: "70942bd8609f2e36d3ae4369ab933717"
server_key_iv: "bc97946c62ea7e6f84bab4cb32320b4f"
server_key_version: "encryptionkey@0"
I20260928 15:45:12.128150 21797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:12.158288 21797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:12.162789 21797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:12.163096 21797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:12.164235 21797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:12.165654 21797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:12.165759 21797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:12.165910 21797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:12.165984 21797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:12.200351 21797 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43631
I20260928 15:45:12.200438 21924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43631 every 8 connection(s)
I20260928 15:45:12.201717 21797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:45:12.204077 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 21797
I20260928 15:45:12.204265 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:45:12.204775 13333 external_mini_cluster.cc:1428] Setting key 5abe01f24ab5041cf984694381b91d3d
Sep 28 15:45:12 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610312, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:45:12.228993 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:12.204923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:57125 (local address 127.13.5.126:43743)
0928 15:45:12.205402 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:12.205408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:12.206702 (+ 1294us) server_negotiation.cc:407] Connection header received
0928 15:45:12.209723 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:12.209732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:12.209820 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:12.209927 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:12.212415 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.213183 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.215767 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.216579 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.216756 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:12.220507 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:12.220516 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:12.220522 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:12.220571 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:12.224726 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:12.225677 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:12.225685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:12.225700 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:12.225934 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:12.226565 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:12.226570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:12.226573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:12.227013 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:12.227206 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:12.228259 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0928 15:45:12.228529 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260928 15:45:12.230162 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:12.205687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:57125)
0928 15:45:12.206502 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:12.206566 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:12.209296 (+ 2730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:12.210270 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:12.210286 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:12.211189 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:12.212168 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:12.212198 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.213374 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.213382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:12.215578 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:12.215592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.216823 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.216828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:12.217070 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:12.217819 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:12.217844 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:12.220234 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:12.224937 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:12.224946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:12.224962 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:12.225504 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:12.226111 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:12.226119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:12.226123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:12.226410 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:12.227176 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:12.227183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:12.227686 (+ 503us) client_negotiation.cc:769] Sending connection context
0928 15:45:12.228121 (+ 435us) client_negotiation.cc:240] Negotiation successful
0928 15:45:12.228612 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:45:12.231979 21925 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43743
I20260928 15:45:12.232363 21925 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:12.233275 21925 heartbeater.cc:511] Master 127.13.5.126:43743 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
I20260928 15:45:12.234978 21601 ts_manager.cc:194] Registered new tserver with Master: 5a9dad2cc18b4a29aac6e59b79ba62ad (127.13.5.66:43631)
I20260928 15:45:12.235836 21601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:57125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:45:12.263821 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43743
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:12.599942 21931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:12.600258 21931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:12.600438 21931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:12.607688 21931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:12.607801 21931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:12.608007 21931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:45:12.618151 21931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:12.619517 21931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:12.620889 21931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:12.635303 21939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:12.635339 21936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:12.635676 21937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:12.641480 21931 server_base.cc:1034] running on GCE node
I20260928 15:45:12.642154 21931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:12.643240 21931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:12.644501 21931 hybrid_clock.cc:648] HybridClock initialized: now 1790610312644447 us; error 43 us; skew 500 ppm
I20260928 15:45:12.644865 21931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:12 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610312, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:12.648224 21931 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:45:12.649614 21931 webserver.cc:460] Webserver started at http://127.13.5.67:41723/ using document root <none> and password file <none>
I20260928 15:45:12.650269 21931 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:12.650369 21931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:12.650636 21931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:12.653607 21931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "32c7085ac0394615b49a2dc783f1d48e"
format_stamp: "Formatted at 2026-09-28 15:45:12 on dist-test-slave-j968"
server_key: "992a894917d226d84d51ba77a9e83d80"
server_key_iv: "8dd466fb4f7f95c8c132e9cdc6f68f61"
server_key_version: "encryptionkey@0"
I20260928 15:45:12.654362 21931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "32c7085ac0394615b49a2dc783f1d48e"
format_stamp: "Formatted at 2026-09-28 15:45:12 on dist-test-slave-j968"
server_key: "992a894917d226d84d51ba77a9e83d80"
server_key_iv: "8dd466fb4f7f95c8c132e9cdc6f68f61"
server_key_version: "encryptionkey@0"
I20260928 15:45:12.659478 21931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:45:12.663031 21946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:12.664597 21931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:12.664745 21931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "32c7085ac0394615b49a2dc783f1d48e"
format_stamp: "Formatted at 2026-09-28 15:45:12 on dist-test-slave-j968"
server_key: "992a894917d226d84d51ba77a9e83d80"
server_key_iv: "8dd466fb4f7f95c8c132e9cdc6f68f61"
server_key_version: "encryptionkey@0"
I20260928 15:45:12.664924 21931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:12.670892 21791 heartbeater.cc:503] Master 127.13.5.126:43743 was elected leader, sending a full tablet report...
I20260928 15:45:12.685992 21931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:12.690641 21931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:12.690943 21931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:12.691996 21931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:12.693498 21931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:12.693573 21931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:12.693739 21931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:12.693789 21931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:12.729888 21931 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:36237
I20260928 15:45:12.729982 22058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:36237 every 8 connection(s)
I20260928 15:45:12.731359 21931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:45:12.737298 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 21931
I20260928 15:45:12.737507 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:45:12.737999 13333 external_mini_cluster.cc:1428] Setting key b300a3633df80cf2677b905d83c217aa
Sep 28 15:45:12 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610312, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:45:12.753080 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:12.734494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:60977 (local address 127.13.5.126:43743)
0928 15:45:12.734995 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:12.735000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:12.735939 (+ 939us) server_negotiation.cc:407] Connection header received
0928 15:45:12.737972 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:12.737978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:12.738084 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:12.738249 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:12.740519 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.741513 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.743140 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.743530 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.743661 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:12.747100 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:12.747105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:12.747108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:12.747140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:12.749794 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:12.750612 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:12.750617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:12.750622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:12.750745 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:12.751169 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:12.751174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:12.751177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:12.751444 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:12.751609 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:12.752447 (+ 838us) server_negotiation.cc:299] Negotiation successful
0928 15:45:12.752668 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":270,"threads_started":1}
I20260928 15:45:12.754129 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:12.735022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:60977)
0928 15:45:12.735751 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:12.735791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:12.737685 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:12.738721 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:12.738735 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:12.739365 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:12.740310 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:12.740336 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.741708 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.741713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:12.742986 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:12.742999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.743768 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.743777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:12.744065 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:12.744685 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:12.744714 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:12.746862 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:12.749971 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:12.749980 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:12.749998 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:12.750420 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:12.750857 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:12.750862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:12.750866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:12.751052 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:12.751571 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:12.751579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:12.752050 (+ 471us) client_negotiation.cc:769] Sending connection context
0928 15:45:12.752336 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 15:45:12.752885 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":282,"threads_started":1}
I20260928 15:45:12.755614 22059 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43743
I20260928 15:45:12.755995 22059 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:12.756728 22059 heartbeater.cc:511] Master 127.13.5.126:43743 requested a full tablet report, sending...
I20260928 15:45:12.758196 21601 ts_manager.cc:194] Registered new tserver with Master: 32c7085ac0394615b49a2dc783f1d48e (127.13.5.67:36237)
I20260928 15:45:12.759059 21601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:60977
I20260928 15:45:12.766247 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab.
Entry for principal oryx/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab.
WARNING: no policy specified for oryx/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab.
Entry for principal oryx/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab.
WARNING: no policy specified for oryx/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab.
Entry for principal oryx/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab.
I20260928 15:45:12.911201 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 21663
Sep 28 15:45:12 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:45:12.929217 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:12.912636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.13.5.65:42141)
0928 15:45:12.913525 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:12.913533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:12.913571 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:45:12.913643 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:12.913650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:12.913924 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:12.914097 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:12.915631 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.916704 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.917956 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:12.918526 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:12.918628 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:12.922731 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:12.922743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:12.922758 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:12.922795 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:12.926116 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:12.926855 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:12.926863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:12.926865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:12.927017 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:12.927393 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:12.927397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:12.927399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:12.927827 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:12.928212 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:12.928556 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 15:45:12.928833 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":481,"threads_started":1}
I20260928 15:45:12.930554 21786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:13.015278 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 21663
I20260928 15:45:13.032851 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 21797
Sep 28 15:45:13 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:45:13.049535 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:13.033777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.13.5.66:43631)
0928 15:45:13.034372 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:13.034380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:13.034406 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:45:13.034485 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:13.034491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:13.034711 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:13.034844 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:13.036102 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:13.037132 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:13.038465 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:13.039073 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:13.039179 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:13.043760 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:13.043774 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:13.043792 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:13.043831 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:13.046938 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:13.047664 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:13.047672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:13.047675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:13.047797 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:13.048189 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:13.048192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:13.048194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:13.048540 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:13.048775 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:13.049039 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 15:45:13.049239 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":323,"threads_started":1}
I20260928 15:45:13.050930 21920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:13.134975 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 21797
I20260928 15:45:13.153275 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 21931
Sep 28 15:45:13 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:45:13.170692 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:13.154404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.13.5.67:36237)
0928 15:45:13.155031 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:13.155038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:13.155068 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:45:13.155133 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:13.155139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:13.155352 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:13.155549 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:13.156923 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:13.158015 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:13.159249 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:13.159866 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:13.160044 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:13.164662 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:13.164673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:13.164689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:13.164730 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:13.167661 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:13.168537 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:13.168544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:13.168547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:13.168689 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:13.169073 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:13.169078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:13.169081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:13.169484 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:13.169786 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:13.170102 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 15:45:13.170335 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":307,"threads_started":1}
I20260928 15:45:13.172163 22054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:13.257552 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 21931
I20260928 15:45:13.276651 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 21569
I20260928 15:45:13.277954 21631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:13.384711 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 21569
I20260928 15:45:13.406674 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43743
--webserver_interface=127.13.5.126
--webserver_port=40523
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:13.756230 22077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:13.756547 22077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:13.756742 22077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:13.764683 22077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:45:13.764814 22077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:13.764945 22077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:13.765051 22077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:45:13.765136 22077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:45:13.775801 22077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=40523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:13.777307 22077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:13.778689 22077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:13.791217 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:13.791317 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:13.791287 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:13.792513 22077 server_base.cc:1034] running on GCE node
I20260928 15:45:13.793290 22077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:13.794747 22077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:13.796003 22077 hybrid_clock.cc:648] HybridClock initialized: now 1790610313795958 us; error 35 us; skew 500 ppm
I20260928 15:45:13.796401 22077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:13 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610313, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:13.799732 22077 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:45:13.801298 22077 webserver.cc:460] Webserver started at http://127.13.5.126:40523/ using document root <none> and password file <none>
I20260928 15:45:13.802042 22077 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:13.802204 22077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:13.808022 22077 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:45:13.811036 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:13.812650 22077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:13.812860 22077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "f6c31b41ed014d8a8e3f4d50f37073c1"
format_stamp: "Formatted at 2026-09-28 15:45:10 on dist-test-slave-j968"
server_key: "a8492c07b74e5939c2fb854064cf8942"
server_key_iv: "2ef3830eb30969a086d64d64d4111e7c"
server_key_version: "encryptionkey@0"
I20260928 15:45:13.813987 22077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:13.832839 22077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:13.837662 22077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:13.837970 22077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:13.857805 22077 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43743
I20260928 15:45:13.857826 22143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43743 every 8 connection(s)
I20260928 15:45:13.859434 22077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:45:13.866582 22144 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:45:13.869410 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 22077
I20260928 15:45:13.870520 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:42141
--local_ip_for_outbound_sockets=127.13.5.65
--tserver_master_addrs=127.13.5.126:43743
--webserver_port=34041
--webserver_interface=127.13.5.65
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.5.65
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:45:13.870553 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Bootstrap starting.
I20260928 15:45:13.892143 22144 log.cc:824] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:13.901126 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:45:13.901707 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Bootstrap complete.
I20260928 15:45:13.910816 22144 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } }
I20260928 15:45:13.911638 22144 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c31b41ed014d8a8e3f4d50f37073c1, State: Initialized, Role: FOLLOWER
I20260928 15:45:13.912310 22144 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } }
I20260928 15:45:13.912575 22144 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:45:13.912686 22144 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:45:13.912879 22144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:45:13.916730 22144 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } }
I20260928 15:45:13.917260 22144 leader_election.cc:302] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6c31b41ed014d8a8e3f4d50f37073c1; no voters:
I20260928 15:45:13.918005 22144 leader_election.cc:288] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:45:13.918147 22148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:45:13.919414 22148 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [term 2 LEADER]: Becoming Leader. State: Replica: f6c31b41ed014d8a8e3f4d50f37073c1, State: Running, Role: LEADER
I20260928 15:45:13.920097 22148 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } }
I20260928 15:45:13.920372 22144 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:45:13.922879 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6c31b41ed014d8a8e3f4d50f37073c1. Latest consensus state: current_term: 2 leader_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } } }
I20260928 15:45:13.923324 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:13.923966 22149 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6c31b41ed014d8a8e3f4d50f37073c1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43743 } } }
I20260928 15:45:13.924347 22149 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:13.936854 22154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:45:13.944310 22154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:45:13.946265 22154 catalog_manager.cc:1259] Loaded cluster ID: 8e9df99689794b6caf87ce0f39e30552
I20260928 15:45:13.946774 22154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:45:13.951357 22154 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:45:13.952890 22154 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f6c31b41ed014d8a8e3f4d50f37073c1: Loaded TSK: 0
I20260928 15:45:13.954406 22154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:45:13.954802 22165 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 15:45:14.229321 22146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:14.229645 22146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:14.229813 22146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:14.238006 22146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:14.238152 22146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:14.238345 22146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:45:14.248886 22146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.65.keytab
--principal=oryx/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:42141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=34041
--tserver_master_addrs=127.13.5.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:14.250279 22146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:14.251659 22146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:14.266472 22170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:14.266438 22171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:14.267026 22173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:14.267803 22146 server_base.cc:1034] running on GCE node
I20260928 15:45:14.268484 22146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:14.269490 22146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:14.270730 22146 hybrid_clock.cc:648] HybridClock initialized: now 1790610314270685 us; error 37 us; skew 500 ppm
I20260928 15:45:14.271070 22146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610314, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:14.281879 22146 init.cc:377] Logged in from keytab as oryx/127.13.5.65@KRBTEST.COM (short username oryx)
I20260928 15:45:14.283782 22146 webserver.cc:460] Webserver started at http://127.13.5.65:34041/ using document root <none> and password file <none>
I20260928 15:45:14.284727 22146 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:14.284889 22146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:14.291081 22146 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 15:45:14.294209 22180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:14.295842 22146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:45:14.296085 22146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "ceaf29b3123c4c448d9afda233e12f1d"
format_stamp: "Formatted at 2026-09-28 15:45:11 on dist-test-slave-j968"
server_key: "24fd8e594c751529b4f440dd4ffd1697"
server_key_iv: "b26516ba3d823d08b149f9acdd12ba34"
server_key_version: "encryptionkey@0"
I20260928 15:45:14.297171 22146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:14.313649 22146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:14.318575 22146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:14.318885 22146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:14.320151 22146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:14.321864 22146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:14.322000 22146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:14.322108 22146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:14.322171 22146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:14.359676 22146 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:42141
I20260928 15:45:14.359793 22292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:42141 every 8 connection(s)
I20260928 15:45:14.361370 22146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:45:14.366554 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 22146
I20260928 15:45:14.367211 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:43631
--local_ip_for_outbound_sockets=127.13.5.66
--tserver_master_addrs=127.13.5.126:43743
--webserver_port=45623
--webserver_interface=127.13.5.66
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.5.66
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:14.384389 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:14.365009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:49373)
0928 15:45:14.365859 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:14.365914 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:14.367768 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:14.369607 (+ 1839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:14.369615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:14.370140 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:14.370702 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.370720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.373152 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.373158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.375184 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.375204 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.376519 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.376529 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.376887 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.377805 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:14.377844 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:14.382701 (+ 4857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":376,"threads_started":1}
W20260928 15:45:14.384521 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:14.364891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:49373 (local address 127.13.5.126:43743)
0928 15:45:14.365727 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:14.365738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:14.366056 (+ 318us) server_negotiation.cc:407] Connection header received
0928 15:45:14.368479 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:14.368504 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:14.369007 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:14.369427 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:14.370923 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.372942 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.375373 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.376251 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.376416 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.383165 (+ 6749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:49373: BlockingRecv error: recv got EOF from 127.13.5.65:49373 (error 108)
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":277,"threads_started":1}
W20260928 15:45:14.385355 22293 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:43743 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.5.126:43743: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:14.394457 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:14.386131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:59089)
0928 15:45:14.386292 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:14.386311 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:14.386532 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:14.386879 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:14.386884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:14.387276 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:14.387442 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.387451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.388585 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.388591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.389503 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.389516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.390224 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.390230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.390415 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.391100 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:14.391118 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:14.394081 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 15:45:14.394517 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:14.386383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:59089 (local address 127.13.5.126:43743)
0928 15:45:14.386588 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:14.386596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:14.386617 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:45:14.386681 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:14.386684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:14.386744 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:14.386847 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:14.387594 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.388435 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.389654 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.390066 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.390198 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.394310 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:59089: BlockingRecv error: recv got EOF from 127.13.5.65:59089 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:14.405232 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:14.395464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:58839)
0928 15:45:14.395667 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:14.395688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:14.395798 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:14.396180 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:14.396186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:14.396590 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:14.396796 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.396808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.397966 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.397972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.399615 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.399631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.400393 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.400400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.400656 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.401304 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:14.401328 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:14.404897 (+ 3569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 15:45:14.405345 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:14.395488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:58839 (local address 127.13.5.126:43743)
0928 15:45:14.395709 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:14.395713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:14.395735 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:45:14.395958 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:14.395962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:14.396039 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:14.396119 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:14.396987 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.397810 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.399770 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.400240 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.400335 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.405107 (+ 4772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:58839: BlockingRecv error: recv got EOF from 127.13.5.65:58839 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260928 15:45:14.405673 22293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:45:14.711076 22297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:14.711444 22297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:14.711655 22297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:14.719326 22297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:14.719434 22297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:14.719609 22297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:45:14.729748 22297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.66.keytab
--principal=oryx/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:43631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=45623
--tserver_master_addrs=127.13.5.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:14.731076 22297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:14.732581 22297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:14.746805 22303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:14.746802 22305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:14.747074 22302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:14.747642 22297 server_base.cc:1034] running on GCE node
I20260928 15:45:14.748296 22297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:14.749202 22297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:14.750500 22297 hybrid_clock.cc:648] HybridClock initialized: now 1790610314750431 us; error 64 us; skew 500 ppm
I20260928 15:45:14.750842 22297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610314, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:14.754166 22297 init.cc:377] Logged in from keytab as oryx/127.13.5.66@KRBTEST.COM (short username oryx)
I20260928 15:45:14.755774 22297 webserver.cc:460] Webserver started at http://127.13.5.66:45623/ using document root <none> and password file <none>
I20260928 15:45:14.756526 22297 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:14.756688 22297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:14.762357 22297 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 15:45:14.765324 22312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:14.766891 22297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:14.767096 22297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "5a9dad2cc18b4a29aac6e59b79ba62ad"
format_stamp: "Formatted at 2026-09-28 15:45:12 on dist-test-slave-j968"
server_key: "70942bd8609f2e36d3ae4369ab933717"
server_key_iv: "bc97946c62ea7e6f84bab4cb32320b4f"
server_key_version: "encryptionkey@0"
I20260928 15:45:14.768189 22297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:14.785990 22297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:14.791044 22297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:14.791324 22297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:14.792375 22297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:14.793864 22297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:14.793982 22297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:14.794109 22297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:14.794198 22297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:14.828352 22297 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43631
I20260928 15:45:14.828454 22424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43631 every 8 connection(s)
I20260928 15:45:14.829922 22297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:45:14.834468 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 22297
I20260928 15:45:14.835256 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:36237
--local_ip_for_outbound_sockets=127.13.5.67
--tserver_master_addrs=127.13.5.126:43743
--webserver_port=41723
--webserver_interface=127.13.5.67
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.5.67
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:45:14.849550 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:14.833004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:59017 (local address 127.13.5.126:43743)
0928 15:45:14.833206 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:14.833210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:14.834611 (+ 1401us) server_negotiation.cc:407] Connection header received
0928 15:45:14.837551 (+ 2940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:14.837555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:14.837628 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:14.837753 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:14.840360 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.841193 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.843746 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.844234 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.844378 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.849277 (+ 4899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:59017: BlockingRecv error: recv got EOF from 127.13.5.66:59017 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:45:14.850322 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:14.833449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:59017)
0928 15:45:14.834339 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:14.834407 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:14.837179 (+ 2772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:14.838016 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:14.838030 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:14.838822 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:14.839841 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.840039 (+ 198us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.841385 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.841394 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.843589 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.843607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.844473 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.844482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.844750 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.845468 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:14.845492 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:14.849103 (+ 3611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":282,"threads_started":1}
W20260928 15:45:14.851034 22425 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:43743 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.5.126:43743: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:14.860453 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:14.851698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:42853)
0928 15:45:14.851928 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:14.851944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:14.852040 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:14.852394 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:14.852401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:14.852750 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:14.852886 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.852894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.854442 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.854445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.855338 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.855349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.856466 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.856470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.856607 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.857280 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:14.857308 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:14.860181 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 15:45:14.860764 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:14.851801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:42853 (local address 127.13.5.126:43743)
0928 15:45:14.852049 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:14.852057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:14.852080 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:45:14.852143 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:14.852151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:14.852236 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:14.852346 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:14.853041 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.854296 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.855486 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.856310 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.856473 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.860506 (+ 4033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:42853: BlockingRecv error: recv got EOF from 127.13.5.66:42853 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:14 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:14.869388 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:14.861599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:55145)
0928 15:45:14.861872 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:14.861890 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:14.862008 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:14.862352 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:14.862357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:14.862726 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:14.862879 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.862887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.863979 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.863984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.864876 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:14.864886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.865606 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.865612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:14.865809 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.866392 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:14.866408 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:14.869089 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160}
W20260928 15:45:14.869550 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:14.861727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:55145 (local address 127.13.5.126:43743)
0928 15:45:14.861875 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:14.861881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:14.861970 (+ 89us) server_negotiation.cc:407] Connection header received
0928 15:45:14.862141 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:14.862146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:14.862212 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:14.862317 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:14.863010 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.863815 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.865022 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:14.865446 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:14.865594 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:14.869292 (+ 3698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:55145: BlockingRecv error: recv got EOF from 127.13.5.66:55145 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 15:45:14.869949 22425 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:45:15.176465 22428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:15.176745 22428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:15.176910 22428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:15.184397 22428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:15.184510 22428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:15.184688 22428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:45:15.195101 22428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/krb5kdc/oryx_127.13.5.67.keytab
--principal=oryx/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:36237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=41723
--tserver_master_addrs=127.13.5.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:15.196590 22428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:15.197995 22428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:15.212213 22433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:15.212143 22436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:15.212239 22434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:15.213106 22428 server_base.cc:1034] running on GCE node
I20260928 15:45:15.213837 22428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:15.215037 22428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:15.216302 22428 hybrid_clock.cc:648] HybridClock initialized: now 1790610315216205 us; error 85 us; skew 500 ppm
I20260928 15:45:15.216667 22428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610315, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:15.220664 22428 init.cc:377] Logged in from keytab as oryx/127.13.5.67@KRBTEST.COM (short username oryx)
I20260928 15:45:15.222303 22428 webserver.cc:460] Webserver started at http://127.13.5.67:41723/ using document root <none> and password file <none>
I20260928 15:45:15.223055 22428 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:15.223217 22428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:15.229347 22428 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:45:15.232468 22443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:15.233942 22428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:45:15.234126 22428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "32c7085ac0394615b49a2dc783f1d48e"
format_stamp: "Formatted at 2026-09-28 15:45:12 on dist-test-slave-j968"
server_key: "992a894917d226d84d51ba77a9e83d80"
server_key_iv: "8dd466fb4f7f95c8c132e9cdc6f68f61"
server_key_version: "encryptionkey@0"
I20260928 15:45:15.235152 22428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:15.250486 22428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:15.254966 22428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:15.255252 22428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:15.256294 22428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:15.257745 22428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:15.257869 22428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:15.257998 22428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:15.258085 22428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:15.294322 22428 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:36237
I20260928 15:45:15.294394 22555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:36237 every 8 connection(s)
I20260928 15:45:15.295822 22428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:45:15.301680 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 22428
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:15.316287 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:15.299206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:38673)
0928 15:45:15.300096 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:15.300154 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:15.302026 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:15.303128 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:15.303137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:15.304022 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:15.304813 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.304831 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.305878 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.305884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.307205 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.307219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.309169 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.309176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.309443 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.310212 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:15.310237 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:15.314839 (+ 4602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
W20260928 15:45:15.316624 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:15.298695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:38673 (local address 127.13.5.126:43743)
0928 15:45:15.298922 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:15.298929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:15.300270 (+ 1341us) server_negotiation.cc:407] Connection header received
0928 15:45:15.302594 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:15.302601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:15.302713 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:15.302889 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:15.304982 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.305709 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.307380 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.308907 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.309057 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.316309 (+ 7252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:38673: BlockingRecv error: recv got EOF from 127.13.5.67:38673 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 15:45:15.317543 22556 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:43743 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.5.126:43743: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
I20260928 15:45:15.325476 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:15.302958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.13.5.126:43743)
0928 15:45:15.303510 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:15.303517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:15.303925 (+ 408us) server_negotiation.cc:407] Connection header received
0928 15:45:15.304111 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:15.304115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:15.304190 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:15.304298 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:15.305877 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.308853 (+ 2976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.310311 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.310813 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.310958 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.314433 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:15.316127 (+ 1694us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:15.316152 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:15.316208 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:15.320717 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:15.321717 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:15.321725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:15.321732 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:15.322168 (+ 436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:15.322658 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:15.322662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:15.322664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:15.323413 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:15.323798 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:15.324725 (+ 927us) server_negotiation.cc:299] Negotiation successful
0928 15:45:15.325095 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:15.328763 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:15.318262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:44313)
0928 15:45:15.318482 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:15.318499 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:15.318698 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:15.319189 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:15.319197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:15.319667 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:15.319898 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.319911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.321335 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.321341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.322654 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.322667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.323711 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.323720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.324028 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.324750 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:15.324782 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:15.328456 (+ 3674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 15:45:15.328944 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:15.318699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:44313 (local address 127.13.5.126:43743)
0928 15:45:15.318891 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:15.318895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:15.318912 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:45:15.318970 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:15.318974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:15.319036 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:15.319116 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:15.320071 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.321173 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.322798 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.323507 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.323625 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.328666 (+ 5041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:44313: BlockingRecv error: recv got EOF from 127.13.5.67:44313 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:15.337872 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:15.329562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:38393)
0928 15:45:15.329930 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:15.329949 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:15.330067 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:15.330403 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:15.330408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:15.330734 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:15.330892 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.330900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.332062 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.332068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.333000 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.333011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.333709 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.333714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.333924 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.334576 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:15.334595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:15.337651 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135}
W20260928 15:45:15.338222 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:15.329660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:38393 (local address 127.13.5.126:43743)
0928 15:45:15.329936 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:15.329944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:15.330004 (+ 60us) server_negotiation.cc:407] Connection header received
0928 15:45:15.330202 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:15.330211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:15.330270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:15.330370 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:15.331039 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.331915 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.333143 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.333558 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.333685 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.337885 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:38393: BlockingRecv error: recv got EOF from 127.13.5.67:38393 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
W20260928 15:45:15.338232 22556 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:15.415189 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:15.406296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:35119)
0928 15:45:15.406814 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:15.406829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:15.406989 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:15.407335 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:15.407340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:15.408017 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:15.408220 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.408231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.409272 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.409284 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.410253 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.410264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.411037 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.411043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.411215 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.411928 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:15.411952 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:15.414944 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
W20260928 15:45:15.415562 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:15.406354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:35119 (local address 127.13.5.126:43743)
0928 15:45:15.406599 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:15.406608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:15.406920 (+ 312us) server_negotiation.cc:407] Connection header received
0928 15:45:15.407121 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:15.407131 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:15.407193 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:15.407296 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:15.408379 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.409123 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.410396 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.410840 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.410991 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.415274 (+ 4283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:35119: BlockingRecv error: recv got EOF from 127.13.5.65:35119 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:15 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:15.879658 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:15.870586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:51247)
0928 15:45:15.871091 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:15.871112 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:15.871233 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:15.871584 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:15.871588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:15.872274 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:15.872505 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.872517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.873684 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.873689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.874730 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:15.874742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.875476 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.875482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:15.875614 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.876324 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:15.876344 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:15.879308 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":275,"threads_started":1}
W20260928 15:45:15.879851 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:15.870684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51247 (local address 127.13.5.126:43743)
0928 15:45:15.871009 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:15.871015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:15.871222 (+ 207us) server_negotiation.cc:407] Connection header received
0928 15:45:15.871331 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:15.871335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:15.871407 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:15.871509 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:15.872678 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.873538 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.874877 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:15.875301 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:15.875394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:15.879577 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:51247: BlockingRecv error: recv got EOF from 127.13.5.66:51247 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 28 15:45:16 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:16 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:16.348188 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:16.339056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:57325)
0928 15:45:16.339632 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:16.339654 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:16.339829 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:16.340280 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:16.340285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:16.340794 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:16.340999 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:16.341009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.342211 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.342216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:16.343189 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:16.343201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.344027 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.344035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:16.344193 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:16.344811 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:16.344829 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:16.347879 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":287,"threads_started":1}
W20260928 15:45:16.348377 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:16.339082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:57325 (local address 127.13.5.126:43743)
0928 15:45:16.339338 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:16.339348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:16.339756 (+ 408us) server_negotiation.cc:407] Connection header received
0928 15:45:16.340055 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:16.340062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:16.340139 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:16.340245 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:16.341163 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.342057 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.343341 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.343773 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.343940 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:16.348162 (+ 4222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:57325: BlockingRecv error: recv got EOF from 127.13.5.67:57325 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 15:45:16 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:16 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:16.425807 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:16.416310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:38331)
0928 15:45:16.417130 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:16.417149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:16.417334 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:16.417901 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:16.417905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:16.418337 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:16.418515 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:16.418524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.419701 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.419706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:16.420874 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:16.420892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.421768 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.421777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:16.421902 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:16.422541 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:16.422559 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:16.425557 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":396,"threads_started":1}
W20260928 15:45:16.426167 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:16.416388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:38331 (local address 127.13.5.126:43743)
0928 15:45:16.416610 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:16.416619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:16.417253 (+ 634us) server_negotiation.cc:407] Connection header received
0928 15:45:16.417618 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:16.417624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:16.417734 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:16.417847 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:16.418666 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.419541 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.421018 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.421515 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.421701 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:16.425803 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:38331: BlockingRecv error: recv got EOF from 127.13.5.65:38331 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 15:45:16 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:16 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:16.889894 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:16.880758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:43185)
0928 15:45:16.881336 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:16.881445 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:16.881625 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:16.881996 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:16.882001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:16.882423 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:16.882573 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:16.882581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.883784 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.883791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:16.885065 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:16.885078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.885867 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.885873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:16.886041 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:16.886681 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:16.886700 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:16.889654 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":324,"threads_started":1}
W20260928 15:45:16.890183 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:16.880833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:43185 (local address 127.13.5.126:43743)
0928 15:45:16.881150 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:16.881158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:16.881538 (+ 380us) server_negotiation.cc:407] Connection header received
0928 15:45:16.881771 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:16.881776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:16.881841 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:16.881956 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:16.882723 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.883649 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.885215 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:16.885708 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:16.885851 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:16.889890 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:43185: BlockingRecv error: recv got EOF from 127.13.5.66:43185 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 28 15:45:17 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:17 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:17.358242 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:17.349261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:40249)
0928 15:45:17.349872 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:17.349931 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:17.350097 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:17.350733 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:17.350738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:17.351196 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:17.351341 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:17.351349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.352499 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.352505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:17.353420 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:17.353431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.354166 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.354172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:17.354336 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:17.355002 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:17.355021 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:17.357988 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":350,"threads_started":1}
W20260928 15:45:17.358567 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:17.349507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:40249 (local address 127.13.5.126:43743)
0928 15:45:17.349738 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:17.349744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:17.350026 (+ 282us) server_negotiation.cc:407] Connection header received
0928 15:45:17.350262 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:17.350266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:17.350330 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:17.350457 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:17.351508 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.352343 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.353562 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.353999 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.354129 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:17.358240 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:40249: BlockingRecv error: recv got EOF from 127.13.5.67:40249 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 15:45:17 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:17 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:17.434818 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:17.426768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:34853)
0928 15:45:17.427327 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:17.427342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:17.427456 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:17.427788 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:17.427792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:17.428226 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:17.428403 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:17.428412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.429444 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.429447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:17.430348 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:17.430359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.430994 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.430998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:17.431139 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:17.431805 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:17.431823 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:17.434527 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":305,"threads_started":1}
W20260928 15:45:17.435154 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:17.426868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:34853 (local address 127.13.5.126:43743)
0928 15:45:17.427067 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:17.427071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:17.427485 (+ 414us) server_negotiation.cc:407] Connection header received
0928 15:45:17.427588 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:17.427592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:17.427654 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:17.427734 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:17.428569 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.429314 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.430487 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.430873 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.430967 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:17.434787 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:34853: BlockingRecv error: recv got EOF from 127.13.5.65:34853 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 15:45:17 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:17 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:17.899041 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:17.890891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:35111)
0928 15:45:17.891465 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:17.891494 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:17.891609 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:17.891983 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:17.891988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:17.892325 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:17.892521 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:17.892529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.893543 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.893547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:17.894437 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:17.894449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.895117 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.895121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:17.895252 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:17.896009 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:17.896028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:17.898763 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
W20260928 15:45:17.899202 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:17.890977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:35111 (local address 127.13.5.126:43743)
0928 15:45:17.891148 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:17.891152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:17.891575 (+ 423us) server_negotiation.cc:407] Connection header received
0928 15:45:17.891719 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:17.891723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:17.891787 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:17.891884 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:17.892650 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.893378 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.894581 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:17.894987 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:17.895086 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:17.898961 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:35111: BlockingRecv error: recv got EOF from 127.13.5.66:35111 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 15:45:18 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:18 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:18.369611 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:18.359322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:59421)
0928 15:45:18.360075 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:18.360098 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:18.360277 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:18.360700 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:18.360706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:18.361201 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:18.361577 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:18.361594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.362717 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.362723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:18.363802 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:18.363816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.364722 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.364727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:18.364863 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:18.365573 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:18.365593 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:18.369207 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":387,"threads_started":1}
W20260928 15:45:18.369747 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:18.359489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:59421 (local address 127.13.5.126:43743)
0928 15:45:18.359727 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:18.359736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:18.360273 (+ 537us) server_negotiation.cc:407] Connection header received
0928 15:45:18.360447 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:18.360451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:18.360544 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:18.360638 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:18.361755 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.362571 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.364019 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.364575 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.364684 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:18.369466 (+ 4782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:59421: BlockingRecv error: recv got EOF from 127.13.5.67:59421 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 15:45:18 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:18 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:18.445689 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:18.435976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:49739)
0928 15:45:18.436498 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:18.436514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:18.436633 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:18.437106 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:18.437111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:18.437618 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:18.437783 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:18.437792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.438950 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.438961 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:18.440090 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:18.440103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.440849 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.440855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:18.441050 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:18.441801 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:18.441822 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:18.445419 (+ 3597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":284,"threads_started":1}
W20260928 15:45:18.446034 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:18.436052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:49739 (local address 127.13.5.126:43743)
0928 15:45:18.436262 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:18.436266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:18.436598 (+ 332us) server_negotiation.cc:407] Connection header received
0928 15:45:18.436780 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:18.436784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:18.436867 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:18.437061 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:18.437968 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.438789 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.440255 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.440686 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.440822 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:18.445714 (+ 4892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:49739: BlockingRecv error: recv got EOF from 127.13.5.65:49739 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 15:45:18 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:18 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:18.909866 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:18.900152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:46873)
0928 15:45:18.900841 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:18.900863 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:18.901062 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:18.901408 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:18.901469 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:18.901883 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:18.902152 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:18.902165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.903593 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.903600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:18.904868 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:18.904885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.905613 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.905618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:18.905767 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:18.906568 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:18.906589 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:18.909628 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":451,"threads_started":1}
W20260928 15:45:18.910167 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:18.900192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:46873 (local address 127.13.5.126:43743)
0928 15:45:18.900390 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:18.900395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:18.900968 (+ 573us) server_negotiation.cc:407] Connection header received
0928 15:45:18.901175 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:18.901179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:18.901244 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:18.901329 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:18.902310 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.903359 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.905022 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:18.905457 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:18.905571 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:18.909864 (+ 4293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:46873: BlockingRecv error: recv got EOF from 127.13.5.66:46873 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 15:45:19 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:19 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:19.386683 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:19.370689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:44401)
0928 15:45:19.371202 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:19.371217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:19.371375 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:19.371803 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:19.371808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:19.372236 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:19.372449 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:19.372458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.373499 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.373506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:19.374496 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:19.374507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.375227 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.375232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:19.375434 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:19.383174 (+ 7740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:19.383197 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:19.386425 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":277,"threads_started":1}
W20260928 15:45:19.386981 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:19.370770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:44401 (local address 127.13.5.126:43743)
0928 15:45:19.371013 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:19.371021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:19.371304 (+ 283us) server_negotiation.cc:407] Connection header received
0928 15:45:19.371584 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:19.371590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:19.371664 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:19.371767 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:19.372593 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.373368 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.374641 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.375081 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.375219 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:19.386712 (+ 11493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:44401: BlockingRecv error: recv got EOF from 127.13.5.67:44401 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 15:45:19 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:19 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:19.455766 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:19.446864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:49771)
0928 15:45:19.447364 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:19.447388 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:19.447499 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:19.447801 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:19.447804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:19.448383 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:19.448562 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:19.448571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.449684 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.449689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:19.450674 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:19.450689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.451396 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.451401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:19.451570 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:19.452231 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:19.452250 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:19.455514 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
W20260928 15:45:19.456058 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:19.446966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:49771 (local address 127.13.5.126:43743)
0928 15:45:19.447249 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:19.447256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:19.447471 (+ 215us) server_negotiation.cc:407] Connection header received
0928 15:45:19.447625 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:19.447629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:19.447700 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:19.447786 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:19.448704 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.449516 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.450820 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.451263 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.451365 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:19.455735 (+ 4370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:49771: BlockingRecv error: recv got EOF from 127.13.5.65:49771 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 28 15:45:19 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:19 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:19.919986 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:19.910960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:41137)
0928 15:45:19.911596 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:19.911617 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:19.911774 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:19.912164 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:19.912170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:19.912626 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:19.912788 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:19.912797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.913893 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.913900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:19.914816 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:19.914827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.915589 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.915593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:19.915734 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:19.916516 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:19.916536 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:19.919711 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":269,"threads_started":1}
W20260928 15:45:19.920392 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:19.911241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:41137 (local address 127.13.5.126:43743)
0928 15:45:19.911474 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:19.911481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:19.911702 (+ 221us) server_negotiation.cc:407] Connection header received
0928 15:45:19.911940 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:19.911945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:19.912028 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:19.912131 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:19.912939 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.913748 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.914965 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:19.915421 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:19.915553 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:19.919981 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:41137: BlockingRecv error: recv got EOF from 127.13.5.66:41137 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 15:45:20 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:20 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:20.396721 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:20.387936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:57413)
0928 15:45:20.388518 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:20.388534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:20.388683 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:20.389056 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:20.389062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:20.389636 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:20.389807 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:20.389816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.390834 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.390840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:20.391765 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:20.391776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.392518 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.392524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:20.392707 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:20.393337 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:20.393356 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:20.396453 (+ 3097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":260,"threads_started":1}
W20260928 15:45:20.397001 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:20.388123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:57413 (local address 127.13.5.126:43743)
0928 15:45:20.388332 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:20.388338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:20.388622 (+ 284us) server_negotiation.cc:407] Connection header received
0928 15:45:20.388818 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:20.388823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:20.388893 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:20.388994 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:20.389956 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.390686 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.391944 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.392363 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.392501 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:20.396685 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:57413: BlockingRecv error: recv got EOF from 127.13.5.67:57413 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 15:45:20 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:20 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:20.465565 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:20.456799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:43027)
0928 15:45:20.457338 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:20.457355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:20.457515 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:20.457850 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:20.457854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:20.458277 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:20.458439 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:20.458451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.459580 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.459586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:20.460521 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:20.460533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.461358 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.461363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:20.461566 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:20.462297 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:20.462315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:20.465344 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":291,"threads_started":1}
W20260928 15:45:20.465816 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:20.456873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:43027 (local address 127.13.5.126:43743)
0928 15:45:20.457144 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:20.457160 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:20.457447 (+ 287us) server_negotiation.cc:407] Connection header received
0928 15:45:20.457639 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:20.457649 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:20.457713 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:20.457813 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:20.458593 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.459444 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.460707 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.461155 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.461291 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:20.465543 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:43027: BlockingRecv error: recv got EOF from 127.13.5.65:43027 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 15:45:20 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:20 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:20.930725 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:20.920957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:56387)
0928 15:45:20.921604 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:20.921627 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:20.921791 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:20.922186 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:20.922191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:20.922632 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:20.922815 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:20.922824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.923984 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.923990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:20.924993 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:20.925006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.926104 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.926111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:20.926313 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:20.927009 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:20.927028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:20.930372 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":322,"threads_started":1}
W20260928 15:45:20.930884 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:20.921002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:56387 (local address 127.13.5.126:43743)
0928 15:45:20.921230 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:20.921239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:20.921710 (+ 471us) server_negotiation.cc:407] Connection header received
0928 15:45:20.921967 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:20.921972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:20.922042 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:20.922153 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:20.923000 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.923795 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.925158 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:20.925876 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:20.926070 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:20.930633 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:56387: BlockingRecv error: recv got EOF from 127.13.5.66:56387 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 15:45:21 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:21 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:21.407064 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:21.397744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:54459)
0928 15:45:21.398345 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:21.398362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:21.398537 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:21.398981 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:21.398986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:21.399396 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:21.399556 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:21.399565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.400737 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.400741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:21.401868 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:21.401880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.402634 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.402641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:21.402809 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:21.403395 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:21.403414 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:21.406742 (+ 3328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":355,"threads_started":1}
W20260928 15:45:21.407266 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:21.397843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:54459 (local address 127.13.5.126:43743)
0928 15:45:21.398136 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:21.398145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:21.398530 (+ 385us) server_negotiation.cc:407] Connection header received
0928 15:45:21.398698 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:21.398702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:21.398794 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:21.398902 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:21.399711 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.400568 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.402027 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.402465 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.402597 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:21.406997 (+ 4400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:54459: BlockingRecv error: recv got EOF from 127.13.5.67:54459 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 28 15:45:21 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:21 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:21.475692 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:21.466713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:37219)
0928 15:45:21.467444 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:21.467462 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:21.467633 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:21.468014 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:21.468019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:21.468552 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:21.468809 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:21.468822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.469834 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.469839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:21.470746 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:21.470757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.471467 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.471473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:21.471638 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:21.472336 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:21.472354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:21.475423 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":349,"threads_started":1}
W20260928 15:45:21.475967 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:21.466818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:37219 (local address 127.13.5.126:43743)
0928 15:45:21.467119 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:21.467131 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:21.467552 (+ 421us) server_negotiation.cc:407] Connection header received
0928 15:45:21.467781 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:21.467786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:21.467860 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:21.467980 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:21.468948 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.469690 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.470886 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.471300 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.471446 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:21.475658 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:37219: BlockingRecv error: recv got EOF from 127.13.5.65:37219 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 15:45:21 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:21 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:21.941411 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:21.931918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:43377)
0928 15:45:21.932550 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:21.932570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:21.932731 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:21.933101 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:21.933106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:21.933576 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:21.933744 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:21.933753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.934840 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.934846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:21.935784 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:21.935796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.936658 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.936664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:21.936864 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:21.937675 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:21.937695 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:21.941010 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":268,"threads_started":1}
W20260928 15:45:21.941433 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:21.932042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:43377 (local address 127.13.5.126:43743)
0928 15:45:21.932320 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:21.932329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:21.932659 (+ 330us) server_negotiation.cc:407] Connection header received
0928 15:45:21.932888 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:21.932892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:21.932956 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:21.933064 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:21.933917 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.934701 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.935990 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:21.936410 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:21.936644 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:21.941159 (+ 4515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:43377: BlockingRecv error: recv got EOF from 127.13.5.66:43377 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 28 15:45:22 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:22 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:22.417342 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:22.408083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:33991)
0928 15:45:22.408732 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:22.408750 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:22.408876 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:22.409196 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:22.409200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:22.409640 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:22.409815 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:22.409824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.411045 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.411049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:22.412105 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:22.412118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.413031 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.413037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:22.413206 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:22.413876 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:22.413896 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:22.417083 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":286,"threads_started":1}
W20260928 15:45:22.417558 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:22.408112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:33991 (local address 127.13.5.126:43743)
0928 15:45:22.408311 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:22.408316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:22.408839 (+ 523us) server_negotiation.cc:407] Connection header received
0928 15:45:22.408990 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:22.408994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:22.409061 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:22.409145 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:22.409955 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.410906 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.412253 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.412817 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.412957 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:22.417323 (+ 4366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:33991: BlockingRecv error: recv got EOF from 127.13.5.67:33991 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 15:45:22 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:22 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:22.486179 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:22.476893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:33965)
0928 15:45:22.477664 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:22.477681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:22.477812 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:22.478126 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:22.478130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:22.478634 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:22.478801 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:22.478810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.479847 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.479851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:22.480968 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:22.480980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.481813 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.481818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:22.481951 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:22.482648 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:22.482667 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:22.485795 (+ 3128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":439,"threads_started":1}
W20260928 15:45:22.486342 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:22.477094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:33965 (local address 127.13.5.126:43743)
0928 15:45:22.477316 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:22.477323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:22.477759 (+ 436us) server_negotiation.cc:407] Connection header received
0928 15:45:22.477917 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:22.477921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:22.477991 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:22.478073 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:22.478946 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.479689 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.481107 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.481638 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.481738 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:22.486041 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:33965: BlockingRecv error: recv got EOF from 127.13.5.65:33965 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 15:45:22 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:22 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:22.951850 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:22.942752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:48895)
0928 15:45:22.943321 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:22.943341 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:22.943512 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:22.943895 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:22.943902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:22.944418 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:22.944574 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:22.944584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.946213 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.946218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:22.947182 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:22.947194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.948006 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.948011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:22.948157 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:22.948752 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:22.948769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:22.951615 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":303,"threads_started":1}
W20260928 15:45:22.952206 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:22.942889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:48895 (local address 127.13.5.126:43743)
0928 15:45:22.943128 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:22.943134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:22.943443 (+ 309us) server_negotiation.cc:407] Connection header received
0928 15:45:22.943650 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:22.943660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:22.943726 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:22.943829 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:22.944757 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.946059 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.947340 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:22.947783 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:22.947943 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:22.951844 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:48895: BlockingRecv error: recv got EOF from 127.13.5.66:48895 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 15:45:23 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:23 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:23.426718 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:23.418326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:57677)
0928 15:45:23.418974 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:23.418991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:23.419108 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:23.419398 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:23.419401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:23.419827 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:23.420007 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:23.420016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.421131 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.421135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:23.422122 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:23.422134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.422862 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.422866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:23.422984 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:23.423611 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:23.423628 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:23.426472 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":413,"threads_started":1}
W20260928 15:45:23.427013 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:23.418403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:57677 (local address 127.13.5.126:43743)
0928 15:45:23.418679 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:23.418688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:23.419065 (+ 377us) server_negotiation.cc:407] Connection header received
0928 15:45:23.419207 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:23.419211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:23.419274 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:23.419352 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:23.420138 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.420989 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.422263 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.422713 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.422806 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:23.426698 (+ 3892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:57677: BlockingRecv error: recv got EOF from 127.13.5.67:57677 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 15:45:23 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:23 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:23.495617 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:23.487236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:40423)
0928 15:45:23.487767 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:23.487783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:23.487970 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:23.488361 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:23.488366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:23.488800 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:23.488968 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:23.488977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.489986 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.489993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:23.490913 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:23.490925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.491645 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.491652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:23.491833 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:23.492457 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:23.492475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:23.495263 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":316,"threads_started":1}
W20260928 15:45:23.495705 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:23.487294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:40423 (local address 127.13.5.126:43743)
0928 15:45:23.487567 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:23.487575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:23.487901 (+ 326us) server_negotiation.cc:407] Connection header received
0928 15:45:23.488135 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:23.488140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:23.488224 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:23.488326 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:23.489119 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.489853 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.491066 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.491483 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.491662 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:23.495463 (+ 3801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:40423: BlockingRecv error: recv got EOF from 127.13.5.65:40423 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 15:45:23 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:23 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:23.961478 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:23.952956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:41875)
0928 15:45:23.953500 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:23.953520 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:23.953674 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:23.954001 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:23.954006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:23.954428 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:23.954592 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:23.954600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.955580 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.955589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:23.956712 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:23.956724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.957431 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.957437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:23.957610 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:23.958181 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:23.958198 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:23.961214 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":311,"threads_started":1}
W20260928 15:45:23.961746 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:23.952999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:41875 (local address 127.13.5.126:43743)
0928 15:45:23.953156 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:23.953161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:23.953603 (+ 442us) server_negotiation.cc:407] Connection header received
0928 15:45:23.953805 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:23.953809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:23.953871 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:23.953967 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:23.954734 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.955457 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.956845 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:23.957276 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:23.957407 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:23.961434 (+ 4027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:41875: BlockingRecv error: recv got EOF from 127.13.5.66:41875 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 15:45:24 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:24 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:24.440244 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:24.427677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:32779)
0928 15:45:24.428238 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:24.428254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:24.428366 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:24.428680 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:24.428683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:24.429054 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:24.429240 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:24.429247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.430208 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.430212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:24.431122 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:24.431132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.431808 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.431812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:24.431982 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:24.436606 (+ 4624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:24.436628 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:24.439797 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":316,"threads_started":1}
W20260928 15:45:24.440608 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:24.427701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:32779 (local address 127.13.5.126:43743)
0928 15:45:24.427990 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:24.427994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:24.428352 (+ 358us) server_negotiation.cc:407] Connection header received
0928 15:45:24.428479 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:24.428483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:24.428550 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:24.428638 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:24.429359 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.430087 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.431238 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.431683 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.431779 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:24.440208 (+ 8429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:32779: BlockingRecv error: recv got EOF from 127.13.5.67:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 15:45:24 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:24 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:24.505205 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:24.496795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:51883)
0928 15:45:24.497318 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:24.497334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:24.497442 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:24.497818 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:24.497822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:24.498226 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:24.498372 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:24.498380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.499404 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.499413 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:24.500531 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:24.500544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.501291 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.501296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:24.501494 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:24.502070 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:24.502087 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:24.504970 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":300,"threads_started":1}
W20260928 15:45:24.505371 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:24.496858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:51883 (local address 127.13.5.126:43743)
0928 15:45:24.497071 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:24.497075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:24.497415 (+ 340us) server_negotiation.cc:407] Connection header received
0928 15:45:24.497578 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:24.497583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:24.497684 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:24.497783 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:24.498518 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.499278 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.500693 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.501141 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.501275 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:24.505133 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:51883: BlockingRecv error: recv got EOF from 127.13.5.65:51883 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 15:45:24 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:24 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:24.971019 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:24.962426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:47453)
0928 15:45:24.962892 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:24.962912 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:24.963025 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:24.963351 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:24.963357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:24.963745 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:24.963960 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:24.963970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.965155 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.965161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:24.966404 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:24.966422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.967118 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.967124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:24.967294 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:24.967906 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:24.967933 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:24.970788 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":274,"threads_started":1}
W20260928 15:45:24.971262 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:24.962539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:47453 (local address 127.13.5.126:43743)
0928 15:45:24.962800 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:24.962806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:24.962995 (+ 189us) server_negotiation.cc:407] Connection header received
0928 15:45:24.963130 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:24.963134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:24.963198 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:24.963280 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:24.964104 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.964992 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.966550 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:24.966970 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:24.967110 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:24.970994 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:47453: BlockingRecv error: recv got EOF from 127.13.5.66:47453 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 15:45:25 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:25 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:25.449415 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:25.441273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:52115)
0928 15:45:25.441909 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:25.441925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:25.442060 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:25.442471 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:25.442476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:25.442967 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:25.443120 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:25.443129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.444203 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.444207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:25.445195 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:25.445208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.445865 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.445871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:25.446003 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:25.446548 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:25.446565 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:25.449179 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":376,"threads_started":1}
W20260928 15:45:25.449635 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:25.441441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:52115 (local address 127.13.5.126:43743)
0928 15:45:25.441660 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:25.441666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:25.442018 (+ 352us) server_negotiation.cc:407] Connection header received
0928 15:45:25.442183 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:25.442187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:25.442298 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:25.442410 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:25.443253 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.444078 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.445323 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.445722 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.445814 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:25.449382 (+ 3568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:52115: BlockingRecv error: recv got EOF from 127.13.5.67:52115 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 15:45:25 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:25 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:25.514833 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:25.506220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:58305)
0928 15:45:25.506750 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:25.506765 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:25.506920 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:25.507274 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:25.507279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:25.507722 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:25.507919 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:25.507928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.508921 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.508928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:25.509818 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:25.509829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.510573 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.510579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:25.510733 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:25.511337 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:25.511355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:25.514482 (+ 3127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":275,"threads_started":1}
W20260928 15:45:25.514932 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:25.506299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:58305 (local address 127.13.5.126:43743)
0928 15:45:25.506564 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:25.506570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:25.506852 (+ 282us) server_negotiation.cc:407] Connection header received
0928 15:45:25.507041 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:25.507047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:25.507139 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:25.507237 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:25.508080 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.508791 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.509955 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.510409 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.510533 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:25.514686 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:58305: BlockingRecv error: recv got EOF from 127.13.5.65:58305 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 15:45:25 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:25 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:25.981539 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:25.972088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:41077)
0928 15:45:25.972809 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:25.972830 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:25.973020 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:25.973561 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:25.973566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:25.974121 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:25.974372 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:25.974384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.975495 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.975501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:25.976487 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:25.976499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.977225 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.977231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:25.977410 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:25.978055 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:25.978076 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:25.981256 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":465,"threads_started":1}
W20260928 15:45:25.981878 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:25.972323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:41077 (local address 127.13.5.126:43743)
0928 15:45:25.972539 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:25.972546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:25.972945 (+ 399us) server_negotiation.cc:407] Connection header received
0928 15:45:25.973186 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:25.973190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:25.973263 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:25.973528 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:25.974510 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.975347 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.976628 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:25.977078 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:25.977211 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:25.981560 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:41077: BlockingRecv error: recv got EOF from 127.13.5.66:41077 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 15:45:26 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:26 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:26.459375 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:26.450414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:54159)
0928 15:45:26.450978 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:26.450995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:26.451134 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:26.451661 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:26.451665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:26.452270 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:26.452480 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:26.452493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.453752 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.453755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:26.454632 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:26.454644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.455401 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.455405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:26.455517 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:26.456248 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:26.456266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:26.459132 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":325,"threads_started":1}
W20260928 15:45:26.459623 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:26.450596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:54159 (local address 127.13.5.126:43743)
0928 15:45:26.450771 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:26.450776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:26.451103 (+ 327us) server_negotiation.cc:407] Connection header received
0928 15:45:26.451252 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:26.451257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:26.451339 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:26.451539 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:26.452653 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.453634 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.454787 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.455256 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.455390 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:26.459374 (+ 3984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:54159: BlockingRecv error: recv got EOF from 127.13.5.67:54159 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 15:45:26 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:26 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:26.525272 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:26.515844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:47129)
0928 15:45:26.516466 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:26.516485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:26.516659 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:26.517088 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:26.517094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:26.517607 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:26.517772 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:26.517781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.518971 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.518977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:26.520200 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:26.520213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.521122 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.521128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:26.521317 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:26.521969 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:26.521987 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:26.525016 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
W20260928 15:45:26.525665 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:26.515951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:47129 (local address 127.13.5.126:43743)
0928 15:45:26.516190 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:26.516199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:26.516577 (+ 378us) server_negotiation.cc:407] Connection header received
0928 15:45:26.516798 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:26.516804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:26.516937 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:26.517047 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:26.517952 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.518778 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.520346 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.520951 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.521112 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:26.525251 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:47129: BlockingRecv error: recv got EOF from 127.13.5.65:47129 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 15:45:26 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:26 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:45:26.994516 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:26.982778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:42647 (local address 127.13.5.126:43743)
0928 15:45:26.983063 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:26.983071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:26.983445 (+ 374us) server_negotiation.cc:407] Connection header received
0928 15:45:26.983672 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:26.983676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:26.983748 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:26.983846 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:26.985029 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.985843 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.988100 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.988649 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.988763 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:26.994310 (+ 5547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:42647: BlockingRecv error: recv got EOF from 127.13.5.66:42647 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 15:45:26.994527 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:26.982697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:42647)
0928 15:45:26.983319 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:26.983349 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:26.983507 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:26.983930 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:26.983938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:26.984572 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:26.984819 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:26.984834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.986006 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.986015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:26.987938 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:26.987958 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:26.988812 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:26.988820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:26.989071 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:26.989858 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:26.989888 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:26.994109 (+ 4221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":308,"threads_started":1}
Sep 28 15:45:27 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:27 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:27.469115 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:27.460411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:59925)
0928 15:45:27.460985 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:27.461000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:27.461174 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:27.461509 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:27.461513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:27.461936 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:27.462109 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:27.462117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.463158 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.463164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:27.464173 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:27.464184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.464965 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.464972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:27.465150 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:27.465814 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:27.465832 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:27.468794 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":316,"threads_started":1}
W20260928 15:45:27.469353 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:27.460577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:59925 (local address 127.13.5.126:43743)
0928 15:45:27.460815 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:27.460823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:27.461108 (+ 285us) server_negotiation.cc:407] Connection header received
0928 15:45:27.461295 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:27.461304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:27.461376 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:27.461476 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:27.462251 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.463017 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.464307 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.464750 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.464902 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:27.469075 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:59925: BlockingRecv error: recv got EOF from 127.13.5.67:59925 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 15:45:27 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:27 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:27.535172 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:27.526161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:45245)
0928 15:45:27.526728 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:27.526746 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:27.526885 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:27.527189 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:27.527193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:27.527640 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:27.527815 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:27.527825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.528965 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.528969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:27.530063 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:27.530076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.530991 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.530999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:27.531165 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:27.531969 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:27.531995 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:27.534939 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":261,"threads_started":1}
W20260928 15:45:27.535526 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:27.526344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:45245 (local address 127.13.5.126:43743)
0928 15:45:27.526689 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:27.526693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:27.526825 (+ 132us) server_negotiation.cc:407] Connection header received
0928 15:45:27.526993 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:27.526997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:27.527061 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:27.527139 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:27.528023 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.528774 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.530213 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.530792 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.530913 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:27.535146 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:45245: BlockingRecv error: recv got EOF from 127.13.5.65:45245 (error 108)
Metrics: {"server-negotiator.queue_time_us":202}
Sep 28 15:45:27 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:27 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:28.004756 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:27.995831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:60779)
0928 15:45:27.996499 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:27.996520 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:27.996646 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:27.997143 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:27.997148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:27.997576 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:27.997739 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:27.997748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.998953 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:27.998961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:27.999933 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:27.999947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.000706 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.000714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:28.000883 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:28.001549 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:28.001566 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:28.004519 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":346,"threads_started":1}
W20260928 15:45:28.004980 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:27.996029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:60779 (local address 127.13.5.126:43743)
0928 15:45:27.996200 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:27.996204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:27.996596 (+ 392us) server_negotiation.cc:407] Connection header received
0928 15:45:27.996779 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:27.996784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:27.996882 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:27.997043 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:27.997881 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:27.998816 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.000083 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.000491 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.000644 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:28.004747 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:60779: BlockingRecv error: recv got EOF from 127.13.5.66:60779 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 15:45:28 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:28 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:28.479547 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:28.470083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:39305)
0928 15:45:28.470649 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:28.470665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:28.470780 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:28.471157 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:28.471160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:28.471622 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:28.471820 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:28.471835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.472981 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.472999 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:28.474451 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:28.474468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.475270 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.475277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:28.475458 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:28.476165 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:28.476185 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:28.479171 (+ 2986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":321,"threads_started":1}
W20260928 15:45:28.479640 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:28.470221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:39305 (local address 127.13.5.126:43743)
0928 15:45:28.470387 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:28.470391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:28.470741 (+ 350us) server_negotiation.cc:407] Connection header received
0928 15:45:28.470878 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:28.470882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:28.470965 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:28.471109 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:28.472044 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.472772 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.474598 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.475125 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.475255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:28.479371 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:39305: BlockingRecv error: recv got EOF from 127.13.5.67:39305 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 15:45:28 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:28 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:28.544713 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:28.536219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:42323)
0928 15:45:28.536728 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:28.536748 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:28.536932 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:28.537279 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:28.537284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:28.537690 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:28.537861 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:28.537869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.539038 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.539043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:28.540095 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:28.540107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.540779 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.540786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:28.540947 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:28.541647 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:28.541664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:28.544464 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":245,"threads_started":1}
W20260928 15:45:28.545022 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:28.536276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:42323 (local address 127.13.5.126:43743)
0928 15:45:28.536627 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:28.536633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:28.536853 (+ 220us) server_negotiation.cc:407] Connection header received
0928 15:45:28.537067 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:28.537076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:28.537155 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:28.537247 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:28.538022 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.538903 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.540233 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:28.540639 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:28.540766 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:28.544655 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:42323: BlockingRecv error: recv got EOF from 127.13.5.65:42323 (error 108)
Metrics: {"server-negotiator.queue_time_us":212}
Sep 28 15:45:29 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:29 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:29.014310 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:29.005778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:50867)
0928 15:45:29.006516 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:29.006538 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:29.006652 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:29.006945 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:29.006948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:29.007357 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:29.007557 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:29.007567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.008711 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.008715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:29.009622 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:29.009632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.010332 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.010335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:29.010451 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:29.011035 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:29.011053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:29.013944 (+ 2891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":456,"threads_started":1}
W20260928 15:45:29.014498 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:29.005845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:50867 (local address 127.13.5.126:43743)
0928 15:45:29.006094 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:29.006105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:29.006621 (+ 516us) server_negotiation.cc:407] Connection header received
0928 15:45:29.006751 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:29.006754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:29.006823 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:29.006915 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:29.007690 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.008589 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.009775 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.010196 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.010297 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:29.014179 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:50867: BlockingRecv error: recv got EOF from 127.13.5.66:50867 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 15:45:29 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:29 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.67@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:29.499218 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:29.480672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.67:56171)
0928 15:45:29.481200 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:29.481217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:29.481334 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:29.481746 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:29.481749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:29.482100 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:29.482285 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:29.482293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.483443 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.483449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:29.484538 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:29.484550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.485296 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.485301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:29.485475 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:29.495780 (+ 10305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:29.495804 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:29.498976 (+ 3172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
W20260928 15:45:29.499471 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:29.480816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:56171 (local address 127.13.5.126:43743)
0928 15:45:29.481005 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:29.481009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:29.481290 (+ 281us) server_negotiation.cc:407] Connection header received
0928 15:45:29.481542 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:29.481546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:29.481613 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:29.481700 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:29.482484 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.483275 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.484698 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.485151 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.485281 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:29.499224 (+ 13943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.67:56171: BlockingRecv error: recv got EOF from 127.13.5.67:56171 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 15:45:29 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:29 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.65@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:29.554555 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:29.545891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.65:54427)
0928 15:45:29.546512 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:29.546534 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:29.546658 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:29.546959 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:29.546963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:29.547440 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:29.547648 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:29.547656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.548791 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.548796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:29.549809 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:29.549822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.550585 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.550589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:29.550714 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:29.551385 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:29.551403 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:29.554283 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":336,"threads_started":1}
W20260928 15:45:29.554804 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:29.546038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:54427 (local address 127.13.5.126:43743)
0928 15:45:29.546278 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:29.546285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:29.546635 (+ 350us) server_negotiation.cc:407] Connection header received
0928 15:45:29.546760 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:29.546764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:29.546826 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:29.546904 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:29.547820 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.548640 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.549970 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:29.550431 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:29.550543 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:29.554575 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.65:54427: BlockingRecv error: recv got EOF from 127.13.5.65:54427 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 15:45:30 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:45:30 dist-test-slave-j968 krb5kdc[21556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.5.66@KRBTEST.COM for oryx/127.13.5.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:45:30.024539 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:30.015510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43743 (local address 127.13.5.66:34795)
0928 15:45:30.016129 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:30.016155 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:30.016277 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:30.016628 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:30.016632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:30.017025 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:30.017233 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:30.017241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.018496 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.018500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:30.019499 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:30.019515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.020307 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.020311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:30.020428 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:30.021073 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:30.021090 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:30.024164 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.5.126:43743: Server oryx/127.13.5.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":285,"threads_started":1}
W20260928 15:45:30.024674 22295 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:30.015659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:34795 (local address 127.13.5.126:43743)
0928 15:45:30.015917 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:30.015923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:30.016269 (+ 346us) server_negotiation.cc:407] Connection header received
0928 15:45:30.016421 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:30.016425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:30.016493 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:30.016581 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:30.017451 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.018300 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.019638 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.020183 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.020279 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:30.024388 (+ 4109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.5.66:34795: BlockingRecv error: recv got EOF from 127.13.5.66:34795 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:45:30.312295 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22146
I20260928 15:45:30.325899 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:30.313519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.13.5.65:42141)
0928 15:45:30.314090 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:30.314098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:30.314126 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:45:30.314188 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:30.314194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:30.314424 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:30.314590 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:30.316186 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.317227 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.318660 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.319350 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.319495 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:30.322143 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:30.322153 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:30.322169 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:30.322207 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:30.325180 (+ 2973us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.5.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:45:30.325554 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53432: Request ticket server kudu/127.13.5.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":283,"threads_started":1}
W20260928 15:45:30.326031 22606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53432: Request ticket server kudu/127.13.5.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:45:30.325996 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22146: Not authorized: Client connection negotiation failed: client connection to 127.13.5.65:42141: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.5.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:45:30.326232 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 22146
I20260928 15:45:30.346472 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22297
I20260928 15:45:30.359169 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:30.347568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.13.5.66:43631)
0928 15:45:30.348220 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:30.348228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:30.348259 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:45:30.348321 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:30.348327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:30.348545 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:30.348725 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:30.350137 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.351198 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.352690 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.353243 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.353341 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:30.356072 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:30.356081 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:30.356096 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:30.356134 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:30.358684 (+ 2550us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.5.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:45:30.358937 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46300: Request ticket server kudu/127.13.5.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":332,"threads_started":1}
W20260928 15:45:30.359290 22607 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46300: Request ticket server kudu/127.13.5.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:45:30.359534 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22297: Not authorized: Client connection negotiation failed: client connection to 127.13.5.66:43631: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.5.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:45:30.359694 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 22297
I20260928 15:45:30.377238 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22428
I20260928 15:45:30.389778 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:30.378375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.13.5.67:36237)
0928 15:45:30.378982 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:30.378990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:30.379018 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:45:30.379082 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:30.379090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:30.379339 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:30.379498 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:30.380929 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.382042 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.383358 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:30.383960 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:30.384058 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:30.386580 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:30.386588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:30.386603 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:30.386641 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:30.389274 (+ 2633us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.5.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:45:30.389529 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43570: Request ticket server kudu/127.13.5.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":313,"threads_started":1}
W20260928 15:45:30.389899 22608 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43570: Request ticket server kudu/127.13.5.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:45:30.389986 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22428: Not authorized: Client connection negotiation failed: client connection to 127.13.5.67:36237: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.13.5.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:45:30.390177 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 22428
I20260928 15:45:30.407558 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22077
I20260928 15:45:30.408995 22139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:30.528502 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 22077
2026-09-28T15:45:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21346 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T15:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:45:30Z Disabled control of system clock
I20260928 15:45:30.608299 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43755
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 15:45:30.948068 22613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:30.948580 22613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:30.948728 22613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:30.956037 22613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:45:30.956135 22613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:30.956228 22613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:30.956321 22613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:45:30.956403 22613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:45:30.965627 22613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:30.966992 22613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:30.968954 22613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:30.982388 22621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:30.982560 22618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:30.982542 22619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:30.983136 22613 server_base.cc:1034] running on GCE node
I20260928 15:45:30.984074 22613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:30.985673 22613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:30.986931 22613 hybrid_clock.cc:648] HybridClock initialized: now 1790610330986874 us; error 46 us; skew 500 ppm
I20260928 15:45:30.987336 22613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:45:30.989833 22613 webserver.cc:460] Webserver started at http://127.13.5.126:44415/ using document root <none> and password file <none>
I20260928 15:45:30.990625 22613 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:30.990758 22613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:30.991115 22613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:30.994274 22613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "6a05abc8a6004ab4acadb036e4cbc08f"
format_stamp: "Formatted at 2026-09-28 15:45:30 on dist-test-slave-j968"
server_key: "ca87306d629581bdcf20458859cc5664"
server_key_iv: "e3b2f059ce5c90adf391db030aff35ee"
server_key_version: "encryptionkey@0"
I20260928 15:45:30.995112 22613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "6a05abc8a6004ab4acadb036e4cbc08f"
format_stamp: "Formatted at 2026-09-28 15:45:30 on dist-test-slave-j968"
server_key: "ca87306d629581bdcf20458859cc5664"
server_key_iv: "e3b2f059ce5c90adf391db030aff35ee"
server_key_version: "encryptionkey@0"
I20260928 15:45:31.000532 22613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:45:31.004221 22627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:31.005700 22613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:45:31.005903 22613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "6a05abc8a6004ab4acadb036e4cbc08f"
format_stamp: "Formatted at 2026-09-28 15:45:30 on dist-test-slave-j968"
server_key: "ca87306d629581bdcf20458859cc5664"
server_key_iv: "e3b2f059ce5c90adf391db030aff35ee"
server_key_version: "encryptionkey@0"
I20260928 15:45:31.006140 22613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:31.024487 22613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:31.025388 22613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:31.025682 22613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:31.043980 22613 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43755
I20260928 15:45:31.043998 22678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43755 every 8 connection(s)
I20260928 15:45:31.045743 22613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:45:31.049494 22679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:31.050451 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 22613
I20260928 15:45:31.050665 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:45:31.051550 13333 external_mini_cluster.cc:1428] Setting key e0ad1a4748bfab97e50a6fa273e67c4e
I20260928 15:45:31.064239 22679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f: Bootstrap starting.
I20260928 15:45:31.066308 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:31.053775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.13.5.126:43755)
0928 15:45:31.054767 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:31.054778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:31.054820 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:45:31.056467 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:31.056487 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:31.056969 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:31.057296 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:31.058055 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:31.059354 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:31.061490 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:31.062234 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:31.062396 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:31.063684 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:31.063692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:31.063707 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:31.064408 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:31.064555 (+ 147us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:31.064578 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:31.064682 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:31.064913 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 15:45:31.065282 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":355,"threads_started":1}
I20260928 15:45:31.067696 22679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:31.068804 22679 log.cc:824] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:31.071612 22679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f: No bootstrap required, opened a new log
I20260928 15:45:31.079147 22679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a05abc8a6004ab4acadb036e4cbc08f" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43755 } }
I20260928 15:45:31.079562 22679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:31.079674 22679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a05abc8a6004ab4acadb036e4cbc08f, State: Initialized, Role: FOLLOWER
I20260928 15:45:31.080295 22679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a05abc8a6004ab4acadb036e4cbc08f" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43755 } }
I20260928 15:45:31.080541 22679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:45:31.080693 22679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:45:31.080875 22679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:31.083045 22679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a05abc8a6004ab4acadb036e4cbc08f" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43755 } }
I20260928 15:45:31.083575 22679 leader_election.cc:302] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a05abc8a6004ab4acadb036e4cbc08f; no voters:
I20260928 15:45:31.084303 22679 leader_election.cc:288] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:45:31.084431 22684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:31.085275 22684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [term 1 LEADER]: Becoming Leader. State: Replica: 6a05abc8a6004ab4acadb036e4cbc08f, State: Running, Role: LEADER
I20260928 15:45:31.085865 22684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a05abc8a6004ab4acadb036e4cbc08f" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43755 } }
I20260928 15:45:31.086447 22679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:45:31.088886 22686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a05abc8a6004ab4acadb036e4cbc08f. Latest consensus state: current_term: 1 leader_uuid: "6a05abc8a6004ab4acadb036e4cbc08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a05abc8a6004ab4acadb036e4cbc08f" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43755 } } }
I20260928 15:45:31.089093 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a05abc8a6004ab4acadb036e4cbc08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a05abc8a6004ab4acadb036e4cbc08f" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43755 } } }
I20260928 15:45:31.089351 22686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:31.089390 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:31.091236 22693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:45:31.096719 22693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:45:31.105427 22693 catalog_manager.cc:1347] Generated new cluster ID: 5a09405dd5f44dccb03d47f7de3f627d
I20260928 15:45:31.105548 22693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:45:31.138486 22693 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:45:31.139947 22693 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:45:31.149792 22693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a05abc8a6004ab4acadb036e4cbc08f: Generated new TSK 0
I20260928 15:45:31.150863 22693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:45:31.166435 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43755
--builtin_ntp_servers=127.13.5.84:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 15:45:31.510228 22703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:31.510597 22703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:31.510756 22703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:31.517992 22703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:31.518143 22703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:31.518327 22703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:45:31.527547 22703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:31.529084 22703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:31.530956 22703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:31.544368 22708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:31.544461 22709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:31.544418 22711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:31.545421 22703 server_base.cc:1034] running on GCE node
I20260928 15:45:31.546033 22703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:31.546962 22703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:31.548158 22703 hybrid_clock.cc:648] HybridClock initialized: now 1790610331548088 us; error 57 us; skew 500 ppm
I20260928 15:45:31.548449 22703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:45:31.550825 22703 webserver.cc:460] Webserver started at http://127.13.5.65:33033/ using document root <none> and password file <none>
I20260928 15:45:31.551566 22703 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:31.551652 22703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:31.552002 22703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:31.554848 22703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "36417638a7ba44a19907b40918814f49"
format_stamp: "Formatted at 2026-09-28 15:45:31 on dist-test-slave-j968"
server_key: "2245ab23abb64e1a01b0070a18514c7d"
server_key_iv: "ce9533525433c322b87266cf7ce682b3"
server_key_version: "encryptionkey@0"
I20260928 15:45:31.555586 22703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "36417638a7ba44a19907b40918814f49"
format_stamp: "Formatted at 2026-09-28 15:45:31 on dist-test-slave-j968"
server_key: "2245ab23abb64e1a01b0070a18514c7d"
server_key_iv: "ce9533525433c322b87266cf7ce682b3"
server_key_version: "encryptionkey@0"
I20260928 15:45:31.560827 22703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:45:31.564409 22717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:31.566032 22703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:45:31.566218 22703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "36417638a7ba44a19907b40918814f49"
format_stamp: "Formatted at 2026-09-28 15:45:31 on dist-test-slave-j968"
server_key: "2245ab23abb64e1a01b0070a18514c7d"
server_key_iv: "ce9533525433c322b87266cf7ce682b3"
server_key_version: "encryptionkey@0"
I20260928 15:45:31.566463 22703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:31.583004 22703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:31.584007 22703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:31.584302 22703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:31.585297 22703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:31.587025 22703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:31.587163 22703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:31.587292 22703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:31.587378 22703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:31.623682 22703 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:45067
I20260928 15:45:31.623739 22829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:45067 every 8 connection(s)
I20260928 15:45:31.625172 22703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:45:31.633229 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 22703
I20260928 15:45:31.633390 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:45:31.634021 13333 external_mini_cluster.cc:1428] Setting key 086f8109819c64302b9a2d20327b6657
I20260928 15:45:31.637084 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43755
--builtin_ntp_servers=127.13.5.84:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 15:45:31.638821 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:31.628023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:40361 (local address 127.13.5.126:43755)
0928 15:45:31.628503 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:31.628508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:31.629739 (+ 1231us) server_negotiation.cc:407] Connection header received
0928 15:45:31.631547 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:31.631553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:31.631664 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:31.631785 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:31.633089 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:31.633980 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:31.635827 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:31.636317 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:31.636455 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:31.637953 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:31.637959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:31.637962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:31.637981 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:31.638030 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:31.638042 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:31.638148 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:31.638372 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 15:45:31.638477 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260928 15:45:31.639971 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:31.628579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43755 (local address 127.13.5.65:40361)
0928 15:45:31.629546 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:31.629603 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:31.631256 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:31.632067 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:31.632076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:31.632171 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:31.632865 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:31.632894 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:31.634151 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:31.634157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:31.635666 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:31.635680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:31.636548 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:31.636567 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:31.636811 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:31.637611 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:45:31.637614 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:31.637659 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:45:31.637744 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:31.638146 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:31.638158 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:31.638169 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:45:31.638361 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 15:45:31.638801 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":331,"threads_started":1}
I20260928 15:45:31.641835 22830 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43755
I20260928 15:45:31.642169 22830 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:31.643028 22830 heartbeater.cc:511] Master 127.13.5.126:43755 requested a full tablet report, sending...
I20260928 15:45:31.645524 22644 ts_manager.cc:194] Registered new tserver with Master: 36417638a7ba44a19907b40918814f49 (127.13.5.65:45067)
I20260928 15:45:31.647828 22644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.65:40361
W20260928 15:45:31.981109 22834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:31.981411 22834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:31.981503 22834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:31.989014 22834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:31.989118 22834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:31.989224 22834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:45:31.998577 22834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:31.999917 22834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:32.001727 22834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:32.015190 22839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:32.015287 22842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:32.015331 22840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:32.016402 22834 server_base.cc:1034] running on GCE node
I20260928 15:45:32.017118 22834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:32.018118 22834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:32.019325 22834 hybrid_clock.cc:648] HybridClock initialized: now 1790610332019276 us; error 42 us; skew 500 ppm
I20260928 15:45:32.019685 22834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:45:32.028764 22834 webserver.cc:460] Webserver started at http://127.13.5.66:42449/ using document root <none> and password file <none>
I20260928 15:45:32.029700 22834 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:32.029832 22834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:32.030190 22834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:32.033200 22834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "2e8ac03e7df14d2984f9368ef0883018"
format_stamp: "Formatted at 2026-09-28 15:45:32 on dist-test-slave-j968"
server_key: "366032fc5bc59178fbe156f90b0db33b"
server_key_iv: "dbdead1076bda0326af376a69fff8893"
server_key_version: "encryptionkey@0"
I20260928 15:45:32.033996 22834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "2e8ac03e7df14d2984f9368ef0883018"
format_stamp: "Formatted at 2026-09-28 15:45:32 on dist-test-slave-j968"
server_key: "366032fc5bc59178fbe156f90b0db33b"
server_key_iv: "dbdead1076bda0326af376a69fff8893"
server_key_version: "encryptionkey@0"
I20260928 15:45:32.039358 22834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:45:32.043589 22848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:32.045243 22834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:32.045451 22834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "2e8ac03e7df14d2984f9368ef0883018"
format_stamp: "Formatted at 2026-09-28 15:45:32 on dist-test-slave-j968"
server_key: "366032fc5bc59178fbe156f90b0db33b"
server_key_iv: "dbdead1076bda0326af376a69fff8893"
server_key_version: "encryptionkey@0"
I20260928 15:45:32.045689 22834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:32.076241 22834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:32.077219 22834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:32.077522 22834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:32.078541 22834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:32.080255 22834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:32.080391 22834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:32.080524 22834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:32.080614 22834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:32.112955 22834 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:33181
I20260928 15:45:32.113014 22960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:33181 every 8 connection(s)
I20260928 15:45:32.114457 22834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:45:32.124624 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 22834
I20260928 15:45:32.124822 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:45:32.125309 13333 external_mini_cluster.cc:1428] Setting key 1c4a18d671efbb52d1cb7cd321279911
I20260928 15:45:32.127756 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:32.117200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:44991 (local address 127.13.5.126:43755)
0928 15:45:32.117379 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:32.117384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:32.118875 (+ 1491us) server_negotiation.cc:407] Connection header received
0928 15:45:32.120740 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:32.120746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:32.120847 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:32.120955 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:32.122078 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.122816 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.124483 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.124981 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.125125 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:32.126727 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:32.126733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:32.126738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:32.126766 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:32.126837 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:32.126855 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:32.126939 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:32.127249 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 15:45:32.127372 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:45:32.128412 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43755
--builtin_ntp_servers=127.13.5.84:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 15:45:32.128868 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:32.117766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43755 (local address 127.13.5.66:44991)
0928 15:45:32.118709 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:32.118750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:32.120458 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:32.121199 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:32.121208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:32.121298 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:32.121903 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:32.121920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.122963 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.122969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:32.124332 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:32.124344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.125222 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.125229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:32.125539 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:32.126307 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:45:32.126310 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:32.126351 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:45:32.126430 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:32.127007 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:32.127018 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:32.127027 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:45:32.127235 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 15:45:32.127602 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":304,"threads_started":1}
I20260928 15:45:32.130399 22961 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43755
I20260928 15:45:32.130760 22961 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:32.131584 22961 heartbeater.cc:511] Master 127.13.5.126:43755 requested a full tablet report, sending...
I20260928 15:45:32.133370 22644 ts_manager.cc:194] Registered new tserver with Master: 2e8ac03e7df14d2984f9368ef0883018 (127.13.5.66:33181)
I20260928 15:45:32.134748 22644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.66:44991
W20260928 15:45:32.483439 22964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:32.483729 22964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:32.483812 22964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:32.491457 22964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:32.491570 22964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:32.491674 22964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:45:32.500945 22964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:32.502266 22964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:32.504184 22964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:32.518329 22969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:32.518257 22970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:32.518357 22972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:32.518886 22964 server_base.cc:1034] running on GCE node
I20260928 15:45:32.519596 22964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:32.520623 22964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:32.521827 22964 hybrid_clock.cc:648] HybridClock initialized: now 1790610332521771 us; error 58 us; skew 500 ppm
I20260928 15:45:32.522172 22964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:45:32.524622 22964 webserver.cc:460] Webserver started at http://127.13.5.67:42935/ using document root <none> and password file <none>
I20260928 15:45:32.525364 22964 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:32.525471 22964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:32.525835 22964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:32.528715 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "5549c975f0794a32a38df719621136b1"
format_stamp: "Formatted at 2026-09-28 15:45:32 on dist-test-slave-j968"
server_key: "a4e582242cb719d013146a24933356af"
server_key_iv: "fea272a4b796a80703f82469a986196c"
server_key_version: "encryptionkey@0"
I20260928 15:45:32.529500 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "5549c975f0794a32a38df719621136b1"
format_stamp: "Formatted at 2026-09-28 15:45:32 on dist-test-slave-j968"
server_key: "a4e582242cb719d013146a24933356af"
server_key_iv: "fea272a4b796a80703f82469a986196c"
server_key_version: "encryptionkey@0"
I20260928 15:45:32.534906 22964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:45:32.538417 22978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:32.539759 22964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:45:32.539978 22964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "5549c975f0794a32a38df719621136b1"
format_stamp: "Formatted at 2026-09-28 15:45:32 on dist-test-slave-j968"
server_key: "a4e582242cb719d013146a24933356af"
server_key_iv: "fea272a4b796a80703f82469a986196c"
server_key_version: "encryptionkey@0"
I20260928 15:45:32.540261 22964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:32.552289 22964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:32.553074 22964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:32.553308 22964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:32.554245 22964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:32.555677 22964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:32.555763 22964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:32.555833 22964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:32.555969 22964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 15:45:32.589697 22964 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:44461
I20260928 15:45:32.589792 23091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:44461 every 8 connection(s)
I20260928 15:45:32.591198 22964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:45:32.595383 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 22964
I20260928 15:45:32.595578 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:45:32.596213 13333 external_mini_cluster.cc:1428] Setting key 8ecfa80e069d33fa393e400eb9197c85
I20260928 15:45:32.605479 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:32.593869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:42995 (local address 127.13.5.126:43755)
0928 15:45:32.594059 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:32.594063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:32.596091 (+ 2028us) server_negotiation.cc:407] Connection header received
0928 15:45:32.598649 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:32.598655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:32.598742 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:32.598852 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:32.600279 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.601088 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.602699 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.603126 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.603225 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:32.604745 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:32.604749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:32.604753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:32.604772 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:32.604804 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:32.604822 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:32.604893 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:32.605148 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 15:45:32.605228 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:45:32.606587 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:32.594382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43755 (local address 127.13.5.67:42995)
0928 15:45:32.595909 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:32.595964 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:32.598303 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:32.599071 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:32.599080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:32.599188 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:32.600120 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:32.600140 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.601238 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.601243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:32.602564 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:32.602576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.603326 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.603331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:32.603605 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:32.604352 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:45:32.604355 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:32.604403 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:45:32.604541 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:32.604939 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:32.604948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:32.604955 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:45:32.605120 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:45:32.605431 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":435,"threads_started":1}
I20260928 15:45:32.608052 23092 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43755
I20260928 15:45:32.608351 23092 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:32.609095 23092 heartbeater.cc:511] Master 127.13.5.126:43755 requested a full tablet report, sending...
I20260928 15:45:32.610860 22644 ts_manager.cc:194] Registered new tserver with Master: 5549c975f0794a32a38df719621136b1 (127.13.5.67:44461)
I20260928 15:45:32.611768 22644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.67:42995
I20260928 15:45:32.612344 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:45:32.619215 22832 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:32.617545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.13.5.126:43755)
0928 15:45:32.617755 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:32.617759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:32.618252 (+ 493us) server_negotiation.cc:407] Connection header received
0928 15:45:32.618442 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:32.618447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:32.618540 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:32.618635 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:32.619022 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49874: BlockingRecv error: recv got EOF from 127.0.0.1:49874 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:45:32.619976 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22703
I20260928 15:45:32.629159 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:32.621003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.13.5.65:45067)
0928 15:45:32.621552 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:32.621558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:32.621585 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:45:32.621642 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:32.621648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:32.621860 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:32.622026 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:32.622385 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.623606 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.625563 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.626208 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.626314 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:32.627622 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:32.627631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:32.627643 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:32.628311 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:32.628358 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:32.628378 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:32.628487 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:32.628651 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 15:45:32.628766 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
I20260928 15:45:32.630494 22825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:32.717440 22830 heartbeater.cc:503] Master 127.13.5.126:43755 was elected leader, sending a full tablet report...
I20260928 15:45:32.718619 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 22703
W20260928 15:45:32.736810 22629 connection.cc:535] server connection from 127.13.5.65:40361 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:45:32.736989 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22834
I20260928 15:45:32.745122 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:32.737949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.13.5.66:33181)
0928 15:45:32.738523 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:32.738532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:32.738564 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:45:32.738657 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:32.738664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:32.738900 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:32.739041 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:32.739400 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.740482 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.741742 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.742336 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.742446 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:32.743483 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:32.743493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:32.743509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:32.744279 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:32.744329 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:32.744360 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:32.744451 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:32.744603 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 15:45:32.744730 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":290,"threads_started":1}
I20260928 15:45:32.746523 22956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:32.829520 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 22834
I20260928 15:45:32.847151 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22964
I20260928 15:45:32.856177 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:32.848204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.13.5.67:44461)
0928 15:45:32.848879 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:32.848888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:32.848921 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:45:32.848988 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:32.848995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:32.849238 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:32.849442 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:32.849963 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.851100 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.852587 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:32.853228 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:32.853403 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:32.854641 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:32.854650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:32.854664 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:32.855276 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:32.855326 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:32.855346 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:32.855491 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:32.855662 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 15:45:32.855801 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":350,"threads_started":1}
I20260928 15:45:32.857488 23087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:32.942200 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 22964
I20260928 15:45:32.958827 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 22613
I20260928 15:45:32.960155 22674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:33.066540 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 22613
2026-09-28T15:45:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2533 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T15:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:45:33Z Disabled control of system clock
I20260928 15:45:33.141465 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45497
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:45:33.497949 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:33.498250 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:33.498335 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:33.505774 23111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:45:33.505903 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:33.505941 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:33.505991 23111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:45:33.506038 23111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:45:33.515455 23111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:45497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:33.516736 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:33.518493 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:33.531239 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:33.531342 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:33.531417 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:33.532015 23111 server_base.cc:1034] running on GCE node
I20260928 15:45:33.533417 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:33.534950 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:33.536185 23111 hybrid_clock.cc:648] HybridClock initialized: now 1790610333536139 us; error 31 us; skew 500 ppm
I20260928 15:45:33.536551 23111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:45:33.538779 23111 webserver.cc:460] Webserver started at http://127.13.5.126:34557/ using document root <none> and password file <none>
I20260928 15:45:33.539494 23111 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:33.539628 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:33.540033 23111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:33.542979 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "635abf840df54c9580bbb788ab84906a"
format_stamp: "Formatted at 2026-09-28 15:45:33 on dist-test-slave-j968"
server_key: "83ef70c6c64da4bf2c68a1fc1f12927a"
server_key_iv: "2ca02c622c3a677f38a7d9462b3cf47b"
server_key_version: "encryptionkey@0"
I20260928 15:45:33.543807 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "635abf840df54c9580bbb788ab84906a"
format_stamp: "Formatted at 2026-09-28 15:45:33 on dist-test-slave-j968"
server_key: "83ef70c6c64da4bf2c68a1fc1f12927a"
server_key_iv: "2ca02c622c3a677f38a7d9462b3cf47b"
server_key_version: "encryptionkey@0"
I20260928 15:45:33.548965 23111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:45:33.552611 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:33.554009 23111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:45:33.554198 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "635abf840df54c9580bbb788ab84906a"
format_stamp: "Formatted at 2026-09-28 15:45:33 on dist-test-slave-j968"
server_key: "83ef70c6c64da4bf2c68a1fc1f12927a"
server_key_iv: "2ca02c622c3a677f38a7d9462b3cf47b"
server_key_version: "encryptionkey@0"
I20260928 15:45:33.554445 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:33.560237 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:33.561024 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:33.561334 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:33.579010 23111 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:45497
I20260928 15:45:33.579047 23177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:45497 every 8 connection(s)
I20260928 15:45:33.580488 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:45:33.584798 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:33.584909 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 23111
I20260928 15:45:33.585125 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:45:33.585935 13333 external_mini_cluster.cc:1428] Setting key a9c55aecec678e9506428bd63538b850
I20260928 15:45:33.595116 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:33.588158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.13.5.126:45497)
0928 15:45:33.589239 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:33.589249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:33.589292 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:45:33.591540 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:33.591562 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:33.592064 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:33.592450 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:33.593489 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:33.593496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:33.593514 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:33.593528 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:33.593586 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:33.593602 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:33.593829 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:33.593964 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 15:45:33.594207 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":411,"threads_started":1}
I20260928 15:45:33.598847 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a: Bootstrap starting.
I20260928 15:45:33.602051 23178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:33.603196 23178 log.cc:824] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:33.606361 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a: No bootstrap required, opened a new log
I20260928 15:45:33.614243 23178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635abf840df54c9580bbb788ab84906a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45497 } }
I20260928 15:45:33.614634 23178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:33.614729 23178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635abf840df54c9580bbb788ab84906a, State: Initialized, Role: FOLLOWER
I20260928 15:45:33.615381 23178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635abf840df54c9580bbb788ab84906a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45497 } }
I20260928 15:45:33.615621 23178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:45:33.615770 23178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:45:33.615962 23178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:33.618099 23178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635abf840df54c9580bbb788ab84906a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45497 } }
I20260928 15:45:33.618625 23178 leader_election.cc:302] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635abf840df54c9580bbb788ab84906a; no voters:
I20260928 15:45:33.619212 23178 leader_election.cc:288] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:45:33.619552 23183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:33.620361 23183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [term 1 LEADER]: Becoming Leader. State: Replica: 635abf840df54c9580bbb788ab84906a, State: Running, Role: LEADER
I20260928 15:45:33.620954 23183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635abf840df54c9580bbb788ab84906a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45497 } }
I20260928 15:45:33.621630 23178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:45:33.623505 23184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635abf840df54c9580bbb788ab84906a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635abf840df54c9580bbb788ab84906a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45497 } } }
I20260928 15:45:33.623988 23184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:33.624886 23188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:45:33.624747 23185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635abf840df54c9580bbb788ab84906a. Latest consensus state: current_term: 1 leader_uuid: "635abf840df54c9580bbb788ab84906a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635abf840df54c9580bbb788ab84906a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45497 } } }
I20260928 15:45:33.625912 23185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:33.631276 23188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:45:33.640329 23188 catalog_manager.cc:1347] Generated new cluster ID: d20039fde7cb448ba11ead1b52bb3f99
I20260928 15:45:33.640450 23188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:45:33.663698 23188 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:45:33.664189 23188 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:45:33.673269 23188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635abf840df54c9580bbb788ab84906a: Generated new TSK 0
I20260928 15:45:33.674116 23188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:45:33.679325 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45497
--builtin_ntp_servers=127.13.5.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:45:34.024106 23202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:34.024385 23202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:34.024556 23202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:34.032647 23202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:34.032747 23202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:34.032923 23202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:45:34.042469 23202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:34.043795 23202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:34.045691 23202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:34.059391 23207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:34.059333 23208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:34.059362 23210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:34.059897 23202 server_base.cc:1034] running on GCE node
I20260928 15:45:34.060714 23202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:34.061687 23202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:34.062891 23202 hybrid_clock.cc:648] HybridClock initialized: now 1790610334062817 us; error 60 us; skew 500 ppm
I20260928 15:45:34.063192 23202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:45:34.065583 23202 webserver.cc:460] Webserver started at http://127.13.5.65:36361/ using document root <none> and password file <none>
I20260928 15:45:34.066314 23202 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:34.066430 23202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:34.066783 23202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:34.069732 23202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "a2d913e139894084a1cd502f8d752729"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "869b9c6bcb7e891897382bbd2fccec53"
server_key_iv: "3a30386416f65a734ac7f3231f22d9eb"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.070535 23202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "a2d913e139894084a1cd502f8d752729"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "869b9c6bcb7e891897382bbd2fccec53"
server_key_iv: "3a30386416f65a734ac7f3231f22d9eb"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.075707 23202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:45:34.079138 23217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:34.080657 23202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:34.080837 23202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "a2d913e139894084a1cd502f8d752729"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "869b9c6bcb7e891897382bbd2fccec53"
server_key_iv: "3a30386416f65a734ac7f3231f22d9eb"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.081068 23202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:34.086851 23202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:34.087654 23202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:34.088012 23202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:34.088940 23202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:34.090342 23202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:34.090457 23202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:34.090590 23202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:34.090694 23202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:34.124523 23202 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:37765
I20260928 15:45:34.124607 23329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:37765 every 8 connection(s)
I20260928 15:45:34.126189 23202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:45:34.135762 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:34.129069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:32777 (local address 127.13.5.126:45497)
0928 15:45:34.129517 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:34.129521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:34.130800 (+ 1279us) server_negotiation.cc:407] Connection header received
0928 15:45:34.133229 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:34.133237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:34.133309 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:34.133433 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:34.134963 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:34.134969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:34.134973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:34.135001 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:34.135044 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:34.135056 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:34.135192 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:34.135394 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 15:45:34.135471 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":219,"threads_started":1}
I20260928 15:45:34.136293 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 23202
I20260928 15:45:34.136628 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:45:34.136986 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:34.129703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45497 (local address 127.13.5.65:32777)
0928 15:45:34.130634 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:34.130679 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:34.132815 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:34.133700 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:34.133708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:34.133803 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:34.134635 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:45:34.134638 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:34.134677 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:45:34.134792 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:34.135155 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:34.135169 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:34.135177 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:45:34.135485 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 15:45:34.135751 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":273,"threads_started":1}
I20260928 15:45:34.137410 13333 external_mini_cluster.cc:1428] Setting key acb1b641e154a332bd12019705e6c679
I20260928 15:45:34.139300 23330 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45497
I20260928 15:45:34.139657 23330 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:34.140326 23330 heartbeater.cc:511] Master 127.13.5.126:45497 requested a full tablet report, sending...
I20260928 15:45:34.140511 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45497
--builtin_ntp_servers=127.13.5.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 15:45:34.142514 23143 ts_manager.cc:194] Registered new tserver with Master: a2d913e139894084a1cd502f8d752729 (127.13.5.65:37765)
W20260928 15:45:34.474682 23334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:34.474977 23334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:34.475068 23334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:34.482378 23334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:34.482484 23334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:34.482594 23334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:45:34.492018 23334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:34.493387 23334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:34.495148 23334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:34.508658 23339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:34.508742 23342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:34.509204 23334 server_base.cc:1034] running on GCE node
W20260928 15:45:34.508769 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:34.509981 23334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:34.510871 23334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:34.512063 23334 hybrid_clock.cc:648] HybridClock initialized: now 1790610334512018 us; error 27 us; skew 500 ppm
I20260928 15:45:34.512394 23334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:45:34.517951 23334 webserver.cc:460] Webserver started at http://127.13.5.66:46127/ using document root <none> and password file <none>
I20260928 15:45:34.518680 23334 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:34.518800 23334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:34.519142 23334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:34.522153 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "d8e0b8b55d8444589e9732d2c996d973"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "360aae05ce80ab07eb3d82231640c763"
server_key_iv: "14ac3e35395f95e11f8b003601f3bf90"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.522960 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "d8e0b8b55d8444589e9732d2c996d973"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "360aae05ce80ab07eb3d82231640c763"
server_key_iv: "14ac3e35395f95e11f8b003601f3bf90"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.528257 23334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:45:34.531800 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:34.533325 23334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:45:34.533571 23334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "d8e0b8b55d8444589e9732d2c996d973"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "360aae05ce80ab07eb3d82231640c763"
server_key_iv: "14ac3e35395f95e11f8b003601f3bf90"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.533797 23334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:34.539734 23334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:34.540665 23334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:34.541124 23334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:34.542112 23334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:34.543583 23334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:34.543716 23334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:34.543879 23334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:34.544036 23334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 15:45:34.577705 23334 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:36541
I20260928 15:45:34.577757 23461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:36541 every 8 connection(s)
I20260928 15:45:34.579203 23334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:45:34.583710 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 23334
I20260928 15:45:34.583916 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:45:34.584460 13333 external_mini_cluster.cc:1428] Setting key 1c20842fe4aa812dc117a8093c6aed49
I20260928 15:45:34.587584 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45497
--builtin_ntp_servers=127.13.5.84:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 15:45:34.589303 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:34.582427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:33363 (local address 127.13.5.126:45497)
0928 15:45:34.582646 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:34.582653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:34.584236 (+ 1583us) server_negotiation.cc:407] Connection header received
0928 15:45:34.586611 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:34.586617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:34.586681 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:34.586793 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:34.588432 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:34.588442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:34.588449 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:34.588474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:34.588520 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:34.588539 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:34.588667 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:34.588878 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 15:45:34.588949 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:45:34.590310 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:34.582954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45497 (local address 127.13.5.66:33363)
0928 15:45:34.584078 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:34.584123 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:34.586305 (+ 2182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:34.587043 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:34.587053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:34.587175 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:34.588081 (+ 906us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:45:34.588085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:34.588135 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:45:34.588242 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:34.588645 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:34.588655 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:34.588663 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:45:34.588930 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 15:45:34.589183 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":305,"threads_started":1}
I20260928 15:45:34.591820 23462 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45497
I20260928 15:45:34.592181 23462 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:34.592902 23462 heartbeater.cc:511] Master 127.13.5.126:45497 requested a full tablet report, sending...
I20260928 15:45:34.594563 23143 ts_manager.cc:194] Registered new tserver with Master: d8e0b8b55d8444589e9732d2c996d973 (127.13.5.66:36541)
W20260928 15:45:34.924188 23465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:34.924564 23465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:34.924734 23465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:34.932215 23465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:34.932315 23465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:34.932497 23465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:45:34.941812 23465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:34.943163 23465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:34.945004 23465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:34.958503 23471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:34.958593 23470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:34.958557 23473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:34.960218 23465 server_base.cc:1034] running on GCE node
I20260928 15:45:34.960873 23465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:34.961930 23465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:34.963184 23465 hybrid_clock.cc:648] HybridClock initialized: now 1790610334963116 us; error 54 us; skew 500 ppm
I20260928 15:45:34.963559 23465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:45:34.965957 23465 webserver.cc:460] Webserver started at http://127.13.5.67:46721/ using document root <none> and password file <none>
I20260928 15:45:34.966717 23465 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:34.966874 23465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:34.967231 23465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:34.970402 23465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "e451ade70f2e4f7a970756c128c07173"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "63bfb77b97e1f21c339a4a5e43f0614c"
server_key_iv: "633540735a4396477c337b142072c03d"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.971285 23465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "e451ade70f2e4f7a970756c128c07173"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "63bfb77b97e1f21c339a4a5e43f0614c"
server_key_iv: "633540735a4396477c337b142072c03d"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.976944 23465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:45:34.980654 23479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:34.982155 23465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:45:34.982355 23465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "e451ade70f2e4f7a970756c128c07173"
format_stamp: "Formatted at 2026-09-28 15:45:34 on dist-test-slave-j968"
server_key: "63bfb77b97e1f21c339a4a5e43f0614c"
server_key_iv: "633540735a4396477c337b142072c03d"
server_key_version: "encryptionkey@0"
I20260928 15:45:34.982594 23465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:34.988704 23465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:34.989472 23465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:34.989739 23465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:34.990628 23465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:34.992173 23465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:34.992302 23465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:34.992429 23465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:34.992515 23465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:35.026467 23465 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:38389
I20260928 15:45:35.026559 23592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:38389 every 8 connection(s)
I20260928 15:45:35.028110 23465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:45:35.032896 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 23465
I20260928 15:45:35.033078 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:45:35.033516 13333 external_mini_cluster.cc:1428] Setting key 49959d51bdcbd83619b0607469da4b66
I20260928 15:45:35.036991 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:35.030871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:45799 (local address 127.13.5.126:45497)
0928 15:45:35.031119 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:35.031126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:35.032574 (+ 1448us) server_negotiation.cc:407] Connection header received
0928 15:45:35.034517 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:35.034521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:35.034573 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:35.034664 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:35.036304 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:35.036310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:35.036316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:35.036339 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:35.036371 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:35.036383 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:35.036504 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:35.036727 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:45:35.036781 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:45:35.038029 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:35.031429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45497 (local address 127.13.5.67:45799)
0928 15:45:35.032402 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:35.032438 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:35.034268 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:35.034944 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:35.034956 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:35.035077 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:35.035999 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:45:35.036002 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:35.036046 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:45:35.036131 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:35.036479 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:35.036489 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:35.036497 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:45:35.036811 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 15:45:35.037024 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":286,"threads_started":1}
I20260928 15:45:35.039335 23593 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45497
I20260928 15:45:35.039734 23593 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:35.040391 23593 heartbeater.cc:511] Master 127.13.5.126:45497 requested a full tablet report, sending...
I20260928 15:45:35.041828 23143 ts_manager.cc:194] Registered new tserver with Master: e451ade70f2e4f7a970756c128c07173 (127.13.5.67:38389)
I20260928 15:45:35.049208 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:45:35.055486 23332 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:45:35.053677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.13.5.126:45497)
0928 15:45:35.053887 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:35.053892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:35.054515 (+ 623us) server_negotiation.cc:407] Connection header received
0928 15:45:35.054704 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:35.054708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:35.054763 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:35.054859 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:35.055155 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47758: BlockingRecv error: recv got EOF from 127.0.0.1:47758 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:45:35.056133 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 23202
I20260928 15:45:35.059849 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:35.057140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.13.5.65:37765)
0928 15:45:35.057793 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:35.057800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:35.057826 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:45:35.057884 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:35.057890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:35.058124 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:35.058246 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:35.059210 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:35.059217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:35.059228 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:35.059247 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:35.059290 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:35.059305 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:35.059408 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:35.059535 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 15:45:35.059616 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":380,"threads_started":1}
I20260928 15:45:35.061144 23325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:35.142990 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 23202
I20260928 15:45:35.160132 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 23334
I20260928 15:45:35.163833 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:35.161166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.13.5.66:36541)
0928 15:45:35.161695 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:35.161702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:35.161728 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:45:35.161792 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:35.161798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:35.162066 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:35.162198 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:35.163109 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:35.163115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:35.163127 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:35.163145 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:35.163187 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:35.163201 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:35.163306 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:35.163425 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 15:45:35.163551 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":262,"threads_started":1}
I20260928 15:45:35.165139 23457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:35.249001 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 23334
I20260928 15:45:35.266359 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 23465
I20260928 15:45:35.270457 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:35.267416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.13.5.67:38389)
0928 15:45:35.268131 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:35.268139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:35.268171 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:45:35.268242 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:35.268249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:35.268561 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:35.268701 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:35.269739 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:35.269746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:35.269757 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:45:35.269768 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:35.269807 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:45:35.269821 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:35.269957 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:35.270003 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 15:45:35.270119 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":327,"threads_started":1}
I20260928 15:45:35.271708 23588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:35.353847 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 23465
I20260928 15:45:35.370755 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 23111
I20260928 15:45:35.372172 23173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:35.477007 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 23111
2026-09-28T15:45:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2410 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:45:35 dist-test-slave-j968 krb5kdc[23611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:45:35 dist-test-slave-j968 krb5kdc[23611](info): set up 2 sockets
Sep 28 15:45:35 dist-test-slave-j968 krb5kdc[23611](info): commencing operation
krb5kdc: starting...
W20260928 15:45:36.587496 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:45:36 dist-test-slave-j968 krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:45:36.833848 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45113
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:45113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:37.166689 23624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:37.167057 23624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:37.167228 23624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:37.174531 23624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:45:37.174695 23624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:37.174811 23624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:37.174912 23624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:45:37.174996 23624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:45:37.184844 23624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:45113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:37.186281 23624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:37.187662 23624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:37.200055 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:37.200078 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:37.200007 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:37.200858 23624 server_base.cc:1034] running on GCE node
I20260928 15:45:37.201731 23624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:37.203187 23624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:37.204416 23624 hybrid_clock.cc:648] HybridClock initialized: now 1790610337204368 us; error 40 us; skew 500 ppm
I20260928 15:45:37.204782 23624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:37 dist-test-slave-j968 krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610337, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:37.207899 23624 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:45:37.209358 23624 webserver.cc:460] Webserver started at http://127.13.5.126:43107/ using document root <none> and password file <none>
I20260928 15:45:37.210069 23624 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:37.210214 23624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:37.210521 23624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:37.213636 23624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "6bc1af40ced14a0084a1a7843f8babc9"
format_stamp: "Formatted at 2026-09-28 15:45:37 on dist-test-slave-j968"
server_key: "aae9864f39580edd15ad5e83066c87c6"
server_key_iv: "c2af9dfcb42b84fc4796ab5392ecefa1"
server_key_version: "encryptionkey@0"
I20260928 15:45:37.214437 23624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "6bc1af40ced14a0084a1a7843f8babc9"
format_stamp: "Formatted at 2026-09-28 15:45:37 on dist-test-slave-j968"
server_key: "aae9864f39580edd15ad5e83066c87c6"
server_key_iv: "c2af9dfcb42b84fc4796ab5392ecefa1"
server_key_version: "encryptionkey@0"
I20260928 15:45:37.219952 23624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:45:37.223591 23640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:37.225243 23624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:45:37.225458 23624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "6bc1af40ced14a0084a1a7843f8babc9"
format_stamp: "Formatted at 2026-09-28 15:45:37 on dist-test-slave-j968"
server_key: "aae9864f39580edd15ad5e83066c87c6"
server_key_iv: "c2af9dfcb42b84fc4796ab5392ecefa1"
server_key_version: "encryptionkey@0"
I20260928 15:45:37.225683 23624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:37.259465 23624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:37.264030 23624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:37.264338 23624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:37.283600 23624 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:45113
I20260928 15:45:37.283643 23691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:45113 every 8 connection(s)
I20260928 15:45:37.285218 23624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:45:37.286984 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 23624
I20260928 15:45:37.287221 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:45:37.288092 13333 external_mini_cluster.cc:1428] Setting key 80c3ac65137224f73f8774a92c46adec
I20260928 15:45:37.290217 23692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:45:37 dist-test-slave-j968 krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:45:37.305145 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9: Bootstrap starting.
I20260928 15:45:37.308776 23692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:37.310163 23692 log.cc:824] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:37.312857 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:37.290683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.13.5.126:45113)
0928 15:45:37.291514 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:37.291527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:37.291568 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:45:37.293270 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:37.293292 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:37.293686 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:37.294034 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:37.295946 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:37.297311 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:37.299118 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:37.299788 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:37.300047 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:37.305021 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:37.305030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:37.305047 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:37.305088 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:37.308320 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:37.309061 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:37.309071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:37.309087 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:37.309363 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:37.309798 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:37.309804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:37.309807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:37.310559 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:37.310932 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:37.311313 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 15:45:37.311741 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":374,"threads_started":1}
I20260928 15:45:37.313517 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9: No bootstrap required, opened a new log
I20260928 15:45:37.321817 23692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc1af40ced14a0084a1a7843f8babc9" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45113 } }
I20260928 15:45:37.322245 23692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:37.322361 23692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc1af40ced14a0084a1a7843f8babc9, State: Initialized, Role: FOLLOWER
I20260928 15:45:37.322948 23692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc1af40ced14a0084a1a7843f8babc9" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45113 } }
I20260928 15:45:37.323207 23692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:45:37.323355 23692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:45:37.323544 23692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:37.325801 23692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc1af40ced14a0084a1a7843f8babc9" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45113 } }
I20260928 15:45:37.326316 23692 leader_election.cc:302] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bc1af40ced14a0084a1a7843f8babc9; no voters:
I20260928 15:45:37.326942 23692 leader_election.cc:288] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:45:37.327266 23697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:37.328053 23697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [term 1 LEADER]: Becoming Leader. State: Replica: 6bc1af40ced14a0084a1a7843f8babc9, State: Running, Role: LEADER
I20260928 15:45:37.328693 23697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc1af40ced14a0084a1a7843f8babc9" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45113 } }
I20260928 15:45:37.329437 23692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:45:37.331650 23698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bc1af40ced14a0084a1a7843f8babc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc1af40ced14a0084a1a7843f8babc9" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45113 } } }
I20260928 15:45:37.332050 23698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:37.332224 23699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bc1af40ced14a0084a1a7843f8babc9. Latest consensus state: current_term: 1 leader_uuid: "6bc1af40ced14a0084a1a7843f8babc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc1af40ced14a0084a1a7843f8babc9" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45113 } } }
I20260928 15:45:37.332633 23699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:45:37.335090 23703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:45:37.341007 23703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:45:37.351171 23703 catalog_manager.cc:1347] Generated new cluster ID: 9014a0ecb50d4abc8f72ad445d52385e
I20260928 15:45:37.351300 23703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:45:37.374987 23703 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:45:37.376549 23703 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:45:37.387121 23703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9: Generated new TSK 0
I20260928 15:45:37.388134 23703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:45:37.460565 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45113
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:37.803795 23718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:37.804193 23718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:37.804381 23718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:37.811991 23718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:37.812142 23718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:37.812340 23718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:45:37.822628 23718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:37.824132 23718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:37.825539 23718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:37.839881 23723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:37.839819 23726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:37.839919 23724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:37.840704 23718 server_base.cc:1034] running on GCE node
I20260928 15:45:37.841393 23718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:37.842415 23718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:37.843621 23718 hybrid_clock.cc:648] HybridClock initialized: now 1790610337843554 us; error 64 us; skew 500 ppm
I20260928 15:45:37.844030 23718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:37 dist-test-slave-j968 krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610337, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:37.847534 23718 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:45:37.849315 23718 webserver.cc:460] Webserver started at http://127.13.5.65:39923/ using document root <none> and password file <none>
I20260928 15:45:37.850093 23718 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:37.850248 23718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:37.850581 23718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:37.853792 23718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "37eb7e37e27e4c8db99e00f032a71e04"
format_stamp: "Formatted at 2026-09-28 15:45:37 on dist-test-slave-j968"
server_key: "997b87fa956f78d292c222650bf3cd3b"
server_key_iv: "e631faa8cd131dbf877b5976bee58ece"
server_key_version: "encryptionkey@0"
I20260928 15:45:37.854619 23718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "37eb7e37e27e4c8db99e00f032a71e04"
format_stamp: "Formatted at 2026-09-28 15:45:37 on dist-test-slave-j968"
server_key: "997b87fa956f78d292c222650bf3cd3b"
server_key_iv: "e631faa8cd131dbf877b5976bee58ece"
server_key_version: "encryptionkey@0"
I20260928 15:45:37.860131 23718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 15:45:37.863837 23733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:37.865309 23718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:37.865496 23718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "37eb7e37e27e4c8db99e00f032a71e04"
format_stamp: "Formatted at 2026-09-28 15:45:37 on dist-test-slave-j968"
server_key: "997b87fa956f78d292c222650bf3cd3b"
server_key_iv: "e631faa8cd131dbf877b5976bee58ece"
server_key_version: "encryptionkey@0"
I20260928 15:45:37.865713 23718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:37.887038 23718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:37.891320 23718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:37.891731 23718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:37.892866 23718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:37.894415 23718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:37.894518 23718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:37.894706 23718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:37.894784 23718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:37.928507 23718 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43905
I20260928 15:45:37.928553 23845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43905 every 8 connection(s)
I20260928 15:45:37.929999 23718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:45:37.934352 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 23718
I20260928 15:45:37.934574 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:45:37.935292 13333 external_mini_cluster.cc:1428] Setting key b351add0bf4552f8b8e8084f21d9e711
Sep 28 15:45:37 dist-test-slave-j968 krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610337, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:45:37.952693 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:37.932924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:35275 (local address 127.13.5.126:45113)
0928 15:45:37.933382 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:37.933386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:37.934410 (+ 1024us) server_negotiation.cc:407] Connection header received
0928 15:45:37.936855 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:37.936861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:37.936963 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:37.937081 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:37.938732 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:37.939477 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:37.941202 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:37.941623 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:37.941777 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:37.946411 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:37.946416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:37.946420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:37.946459 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:37.949119 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:37.950131 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:37.950135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:37.950137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:37.950260 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:37.950788 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:37.950791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:37.950793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:37.951057 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:37.951184 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:37.952242 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0928 15:45:37.952425 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260928 15:45:37.954341 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:37.933522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45113 (local address 127.13.5.65:35275)
0928 15:45:37.934244 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:37.934288 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:37.936549 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:37.937346 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:37.937356 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:37.937913 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:37.938537 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:37.938556 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:37.939669 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:37.939674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:37.941041 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:37.941054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:37.941896 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:37.941904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:37.942156 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:37.942980 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:37.943017 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:37.946093 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:37.949279 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:37.949292 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:37.949315 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:37.949982 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:37.950377 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:37.950384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:37.950390 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:37.950669 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:37.951203 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:37.951215 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:37.951736 (+ 521us) client_negotiation.cc:769] Sending connection context
0928 15:45:37.952091 (+ 355us) client_negotiation.cc:240] Negotiation successful
0928 15:45:37.952720 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":280,"threads_started":1}
I20260928 15:45:37.956133 23846 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45113
I20260928 15:45:37.956517 23846 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:37.957505 23846 heartbeater.cc:511] Master 127.13.5.126:45113 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:45:37.959995 23657 ts_manager.cc:194] Registered new tserver with Master: 37eb7e37e27e4c8db99e00f032a71e04 (127.13.5.65:43905)
I20260928 15:45:37.961620 23657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:35275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:45:37.991370 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45113
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:38.328724 23852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:38.329012 23852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:38.329161 23852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:38.336767 23852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:38.336884 23852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:38.336998 23852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:45:38.346959 23852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:38.348301 23852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:38.349647 23852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:38.363265 23858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:38.363427 23857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:38.363672 23860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:38.364555 23852 server_base.cc:1034] running on GCE node
I20260928 15:45:38.365181 23852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:38.366132 23852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:38.367379 23852 hybrid_clock.cc:648] HybridClock initialized: now 1790610338367335 us; error 38 us; skew 500 ppm
I20260928 15:45:38.367687 23852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:38 dist-test-slave-j968 krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610338, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:38.371117 23852 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:45:38.372602 23852 webserver.cc:460] Webserver started at http://127.13.5.66:36723/ using document root <none> and password file <none>
I20260928 15:45:38.373330 23852 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:38.373436 23852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:38.373689 23852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:38.376698 23852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "7725019446cf4bb8bf33dcaea89abcf6"
format_stamp: "Formatted at 2026-09-28 15:45:38 on dist-test-slave-j968"
server_key: "396305e26c63d50ff349fa440e69c4fb"
server_key_iv: "a80aeec47889086ac18c0f5d05fdbaff"
server_key_version: "encryptionkey@0"
I20260928 15:45:38.377471 23852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "7725019446cf4bb8bf33dcaea89abcf6"
format_stamp: "Formatted at 2026-09-28 15:45:38 on dist-test-slave-j968"
server_key: "396305e26c63d50ff349fa440e69c4fb"
server_key_iv: "a80aeec47889086ac18c0f5d05fdbaff"
server_key_version: "encryptionkey@0"
I20260928 15:45:38.382963 23852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:45:38.386844 23867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:38.388500 23852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:45:38.388655 23852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "7725019446cf4bb8bf33dcaea89abcf6"
format_stamp: "Formatted at 2026-09-28 15:45:38 on dist-test-slave-j968"
server_key: "396305e26c63d50ff349fa440e69c4fb"
server_key_iv: "a80aeec47889086ac18c0f5d05fdbaff"
server_key_version: "encryptionkey@0"
I20260928 15:45:38.388835 23852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:38.402417 23852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:38.407015 23852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:38.407274 23852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:38.408448 23852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:38.409894 23852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:38.409970 23852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:38.410054 23852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:38.410099 23852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:38.443497 23852 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:32821
I20260928 15:45:38.443517 23982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:32821 every 8 connection(s)
I20260928 15:45:38.444896 23852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:45:38.455097 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 23852
I20260928 15:45:38.455334 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:45:38.455806 13333 external_mini_cluster.cc:1428] Setting key 13492fc84649ff25d963d06e2443eed1
Sep 28 15:45:38 dist-test-slave-j968 krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610338, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:45:38.465548 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:38.447695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:55627 (local address 127.13.5.126:45113)
0928 15:45:38.447948 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:38.447954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:38.449259 (+ 1305us) server_negotiation.cc:407] Connection header received
0928 15:45:38.451039 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:38.451044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:38.451118 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:38.451218 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:38.452916 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:38.453663 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:38.455307 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:38.455830 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:38.456013 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:38.459704 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:38.459709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:38.459712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:38.459741 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:38.462427 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:38.463187 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:38.463192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:38.463198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:38.463326 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:38.463755 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:38.463762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:38.463766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:38.464094 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:38.464246 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:38.464935 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 15:45:38.465165 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:45:38.466543 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:38.448223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45113 (local address 127.13.5.66:55627)
0928 15:45:38.449089 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:38.449139 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:38.450758 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:38.451429 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:38.451437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:38.452068 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:38.452728 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:38.452748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:38.453814 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:38.453818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:38.455150 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:38.455163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:38.456114 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:38.456123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:38.456423 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:38.457182 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:38.457206 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:38.459478 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:38.462604 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:38.462613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:38.462632 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:38.463057 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:38.463440 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:38.463445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:38.463449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:38.463645 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:38.464221 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:38.464229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:38.464572 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 15:45:38.464843 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 15:45:38.465398 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":312,"threads_started":1}
I20260928 15:45:38.468514 23983 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45113
I20260928 15:45:38.468866 23983 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:38.469864 23983 heartbeater.cc:511] Master 127.13.5.126:45113 requested a full tablet report, sending...
I20260928 15:45:38.471633 23657 ts_manager.cc:194] Registered new tserver with Master: 7725019446cf4bb8bf33dcaea89abcf6 (127.13.5.66:32821)
I20260928 15:45:38.472924 23657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:55627
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:45:38.511857 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45113
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:45:38.848788 23988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:45:38.849097 23988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:45:38.849272 23988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:45:38.856693 23988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:45:38.856848 23988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:45:38.857021 23988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:45:38.866804 23988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:45:38.868336 23988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:45:38.869760 23988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:45:38.883474 23994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:38.883456 23996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:45:38.883509 23993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:45:38.884328 23988 server_base.cc:1034] running on GCE node
I20260928 15:45:38.884975 23988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:45:38.885989 23988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:45:38.887293 23988 hybrid_clock.cc:648] HybridClock initialized: now 1790610338887230 us; error 53 us; skew 500 ppm
I20260928 15:45:38.887681 23988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:45:38 dist-test-slave-j968 krb5kdc[23611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610338, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:45:38.890847 23988 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:45:38.892416 23988 webserver.cc:460] Webserver started at http://127.13.5.67:45473/ using document root <none> and password file <none>
I20260928 15:45:38.893136 23988 fs_manager.cc:359] Metadata directory not provided
I20260928 15:45:38.893278 23988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:45:38.893600 23988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:45:38.896687 23988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "4341f72d1c5d4960b55529e71fea0168"
format_stamp: "Formatted at 2026-09-28 15:45:38 on dist-test-slave-j968"
server_key: "77e2d2efe319664d36f6df545ad0ce2d"
server_key_iv: "e9e44486433ecbbc110a8454e437edff"
server_key_version: "encryptionkey@0"
I20260928 15:45:38.897396 23988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "4341f72d1c5d4960b55529e71fea0168"
format_stamp: "Formatted at 2026-09-28 15:45:38 on dist-test-slave-j968"
server_key: "77e2d2efe319664d36f6df545ad0ce2d"
server_key_iv: "e9e44486433ecbbc110a8454e437edff"
server_key_version: "encryptionkey@0"
I20260928 15:45:38.902482 23988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 15:45:38.906095 24003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:38.907547 23988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:45:38.907683 23988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "4341f72d1c5d4960b55529e71fea0168"
format_stamp: "Formatted at 2026-09-28 15:45:38 on dist-test-slave-j968"
server_key: "77e2d2efe319664d36f6df545ad0ce2d"
server_key_iv: "e9e44486433ecbbc110a8454e437edff"
server_key_version: "encryptionkey@0"
I20260928 15:45:38.907859 23988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:45:38.919377 23988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:45:38.923362 23988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:45:38.923676 23988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:45:38.924629 23988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:45:38.925881 23988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:45:38.925961 23988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:38.926050 23988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:45:38.926092 23988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:45:38.960035 23988 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:43885
I20260928 15:45:38.960084 24116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:43885 every 8 connection(s)
I20260928 15:45:38.961370 23988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:45:38.964671 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 23988
I20260928 15:45:38.964890 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:45:38.964938 23846 heartbeater.cc:503] Master 127.13.5.126:45113 was elected leader, sending a full tablet report...
I20260928 15:45:38.965411 13333 external_mini_cluster.cc:1428] Setting key 5dc8f8c5c9334c671cdcf57e70fae407
Sep 28 15:45:38 dist-test-slave-j968 krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610338, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:45:38.982601 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:38.963992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:48341 (local address 127.13.5.126:45113)
0928 15:45:38.964228 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:38.964235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:38.965471 (+ 1236us) server_negotiation.cc:407] Connection header received
0928 15:45:38.968101 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:38.968106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:38.968188 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:38.968297 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:38.970054 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:38.970856 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:38.972496 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:38.972938 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:38.973040 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:38.976427 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:38.976432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:38.976436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:38.976467 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:38.979096 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:38.979931 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:38.979935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:38.979937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:38.980098 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:38.980647 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:38.980652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:38.980656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:38.981060 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:38.981227 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:38.981986 (+ 759us) server_negotiation.cc:299] Negotiation successful
0928 15:45:38.982242 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:45:38.983599 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:38.964440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45113 (local address 127.13.5.67:48341)
0928 15:45:38.965281 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:38.965326 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:38.967781 (+ 2455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:38.968629 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:38.968637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:38.969285 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:45:38.969872 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:38.969890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:38.970998 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:38.971003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:38.972357 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:38.972373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:38.973123 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:38.973128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:38.973319 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:38.973977 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:45:38.973999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:45:38.976231 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:45:38.979233 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:38.979241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:38.979258 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:38.979735 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:38.980250 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:45:38.980255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:45:38.980258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:45:38.980521 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:45:38.981197 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:45:38.981205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:45:38.981602 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 15:45:38.981875 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 15:45:38.982406 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":355,"threads_started":1}
I20260928 15:45:38.985219 24117 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45113
I20260928 15:45:38.985586 24117 heartbeater.cc:464] Registering TS with master...
I20260928 15:45:38.986317 24117 heartbeater.cc:511] Master 127.13.5.126:45113 requested a full tablet report, sending...
I20260928 15:45:38.987748 23656 ts_manager.cc:194] Registered new tserver with Master: 4341f72d1c5d4960b55529e71fea0168 (127.13.5.67:43885)
I20260928 15:45:38.988682 23656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:48341
I20260928 15:45:38.993708 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:45:39.010366 23848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:38.998065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.13.5.126:45113)
0928 15:45:38.998282 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:38.998291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:38.998699 (+ 408us) server_negotiation.cc:407] Connection header received
0928 15:45:38.998832 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:38.998836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:38.998895 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:38.998973 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:39.000413 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.001170 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.002347 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.002779 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.002872 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.005199 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:39.005203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:39.005206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:39.005244 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:39.008023 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:39.008633 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:39.008637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:39.008639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:39.008754 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:39.009081 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:39.009084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:39.009086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:39.009388 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:39.009586 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.009855 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.010010 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:45:39.014731 23656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:45:39.021692 23656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:45:39.049167 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.035453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.13.5.65:43905)
0928 15:45:39.038504 (+ 3051us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.038519 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.038562 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:45:39.038717 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.038729 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.039087 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.039367 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:39.041049 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.042833 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.046120 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.047488 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.047613 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.047848 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.048620 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.048800 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2783,"thread_start_us":2539,"threads_started":1}
I20260928 15:45:39.049221 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.035120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:32821 (local address 127.0.0.1:40568)
0928 15:45:39.040855 (+ 5735us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:39.040873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:39.041033 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:39.041942 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:39.041947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:39.041972 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:39.042526 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.042539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.044380 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.044387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.045819 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.045831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.048013 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.048025 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.048505 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.048545 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:45:39.048629 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:45:39.048829 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5592,"mutex_wait_us":250,"thread_start_us":274,"threads_started":1}
I20260928 15:45:39.049279 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.035326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.13.5.66:32821)
0928 15:45:39.040175 (+ 4849us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.040187 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.041073 (+ 886us) server_negotiation.cc:407] Connection header received
0928 15:45:39.041188 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.041198 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.041473 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.041695 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:39.042706 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.044282 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.045973 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.047488 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.047613 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.047815 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.048700 (+ 885us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.048903 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4650,"thread_start_us":4441,"threads_started":1}
I20260928 15:45:39.050228 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.034870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43905 (local address 127.0.0.1:60378)
0928 15:45:39.035903 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:39.035962 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:39.036172 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:39.040245 (+ 4073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:39.040257 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:39.040298 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:39.040829 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.040849 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.042989 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.043001 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.045956 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.045973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.048037 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.048043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.048340 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.048399 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 15:45:39.049644 (+ 1245us) client_negotiation.cc:240] Negotiation successful
0928 15:45:39.049778 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":25,"thread_start_us":5688,"threads_started":1}
I20260928 15:45:39.051120 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.041992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:43885 (local address 127.0.0.1:54652)
0928 15:45:39.042661 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:39.042680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:39.042814 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:39.043631 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:39.043635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:39.043664 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:39.044110 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.044121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.046186 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.046194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.047835 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.047849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.050253 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.050260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.050490 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.050527 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:45:39.050639 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 15:45:39.050787 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":328,"threads_started":1}
I20260928 15:45:39.051411 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.041301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.13.5.67:43885)
0928 15:45:39.041998 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.042007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.042787 (+ 780us) server_negotiation.cc:407] Connection header received
0928 15:45:39.042971 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.042978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.043197 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.043365 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:39.044357 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.046069 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.048086 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.049681 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.049789 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.050032 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.050700 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.050906 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":360,"threads_started":1}
I20260928 15:45:39.053792 23781 tablet_service.cc:1467] Processing CreateTablet for tablet ea77a82f051346459c099b915d7b86e2 (DEFAULT_TABLE table=test-table [id=4927541c52c746cb98bd7acfd80e709a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:39.053907 23917 tablet_service.cc:1467] Processing CreateTablet for tablet ea77a82f051346459c099b915d7b86e2 (DEFAULT_TABLE table=test-table [id=4927541c52c746cb98bd7acfd80e709a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:39.055130 23781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea77a82f051346459c099b915d7b86e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:39.055393 23917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea77a82f051346459c099b915d7b86e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:39.055364 24052 tablet_service.cc:1467] Processing CreateTablet for tablet ea77a82f051346459c099b915d7b86e2 (DEFAULT_TABLE table=test-table [id=4927541c52c746cb98bd7acfd80e709a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:45:39.056859 24052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea77a82f051346459c099b915d7b86e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:45:39.066504 24135 tablet_bootstrap.cc:492] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Bootstrap starting.
I20260928 15:45:39.070376 24135 tablet_bootstrap.cc:654] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:39.070829 24136 tablet_bootstrap.cc:492] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Bootstrap starting.
I20260928 15:45:39.071960 24135 log.cc:824] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:39.072383 24137 tablet_bootstrap.cc:492] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Bootstrap starting.
I20260928 15:45:39.074582 24136 tablet_bootstrap.cc:654] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:39.075498 24137 tablet_bootstrap.cc:654] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Neither blocks nor log segments found. Creating new log.
I20260928 15:45:39.075835 24136 log.cc:824] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:39.076002 24135 tablet_bootstrap.cc:492] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: No bootstrap required, opened a new log
I20260928 15:45:39.076277 24135 ts_tablet_manager.cc:1397] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260928 15:45:39.076705 24137 log.cc:824] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Log is configured to *not* fsync() on all Append() calls
I20260928 15:45:39.079244 24136 tablet_bootstrap.cc:492] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: No bootstrap required, opened a new log
I20260928 15:45:39.079521 24136 ts_tablet_manager.cc:1397] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:45:39.081190 24137 tablet_bootstrap.cc:492] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: No bootstrap required, opened a new log
I20260928 15:45:39.081470 24137 ts_tablet_manager.cc:1397] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260928 15:45:39.084686 24135 raft_consensus.cc:348] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.085068 24135 raft_consensus.cc:374] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:39.085167 24135 raft_consensus.cc:729] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eb7e37e27e4c8db99e00f032a71e04, State: Initialized, Role: FOLLOWER
I20260928 15:45:39.085794 24135 consensus_queue.cc:260] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.087574 24135 ts_tablet_manager.cc:1428] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260928 15:45:39.090137 24136 raft_consensus.cc:348] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.090585 24136 raft_consensus.cc:374] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:39.090685 24136 raft_consensus.cc:729] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7725019446cf4bb8bf33dcaea89abcf6, State: Initialized, Role: FOLLOWER
I20260928 15:45:39.091322 24136 consensus_queue.cc:260] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.092547 23983 heartbeater.cc:503] Master 127.13.5.126:45113 was elected leader, sending a full tablet report...
I20260928 15:45:39.092919 24137 raft_consensus.cc:348] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.093322 24136 ts_tablet_manager.cc:1428] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260928 15:45:39.093410 24137 raft_consensus.cc:374] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:45:39.093510 24137 raft_consensus.cc:729] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4341f72d1c5d4960b55529e71fea0168, State: Initialized, Role: FOLLOWER
I20260928 15:45:39.094107 24137 consensus_queue.cc:260] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.095278 24117 heartbeater.cc:503] Master 127.13.5.126:45113 was elected leader, sending a full tablet report...
I20260928 15:45:39.095937 24137 ts_tablet_manager.cc:1428] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260928 15:45:39.182729 23847 tablet.cc:2369] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:45:39.197162 23984 tablet.cc:2369] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:45:39.213078 24118 tablet.cc:2369] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:45:39.441053 24142 raft_consensus.cc:482] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:45:39.441403 24142 raft_consensus.cc:504] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.443461 24142 leader_election.cc:288] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37eb7e37e27e4c8db99e00f032a71e04 (127.13.5.65:43905), 4341f72d1c5d4960b55529e71fea0168 (127.13.5.67:43885)
I20260928 15:45:39.452143 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.444741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:59149 (local address 127.13.5.67:43885)
0928 15:45:39.444954 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.444961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.445433 (+ 472us) server_negotiation.cc:407] Connection header received
0928 15:45:39.445713 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.445724 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.445785 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.445880 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:39.446583 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.447588 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.449987 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.450788 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.450945 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.451070 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.451726 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.451854 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:45:39.452206 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.443959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51897 (local address 127.13.5.65:43905)
0928 15:45:39.444185 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.444192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.444548 (+ 356us) server_negotiation.cc:407] Connection header received
0928 15:45:39.445048 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.445057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.445151 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.445228 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:45:39.446142 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.447038 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.449565 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.450833 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.450968 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.451071 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.451795 (+ 724us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.451964 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:45:39.452294 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.444505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:43885 (local address 127.13.5.66:59149)
0928 15:45:39.445325 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:39.445341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:39.445543 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:39.445898 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:39.445902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:39.445931 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:39.446406 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.446420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.447734 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.447739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.449824 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.449834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.451299 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.451309 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.451507 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.451584 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 15:45:39.451696 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 15:45:39.451833 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":49,"spinlock_wait_cycles":12416,"thread_start_us":513,"threads_started":1}
I20260928 15:45:39.453176 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.443693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43905 (local address 127.13.5.66:51897)
0928 15:45:39.444426 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 15:45:39.444447 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:45:39.444666 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:45:39.445445 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:45:39.445449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:45:39.445516 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:45:39.445968 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.445981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.447176 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.447182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.449385 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:45:39.449403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.451243 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.451250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:45:39.451611 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.451646 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:45:39.452636 (+ 990us) client_negotiation.cc:240] Negotiation successful
0928 15:45:39.452789 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":46,"spinlock_wait_cycles":6784,"thread_start_us":387,"threads_started":1}
I20260928 15:45:39.453269 24072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea77a82f051346459c099b915d7b86e2" candidate_uuid: "7725019446cf4bb8bf33dcaea89abcf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4341f72d1c5d4960b55529e71fea0168" is_pre_election: true
I20260928 15:45:39.453889 24072 raft_consensus.cc:2454] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7725019446cf4bb8bf33dcaea89abcf6 in term 0.
I20260928 15:45:39.454241 23801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea77a82f051346459c099b915d7b86e2" candidate_uuid: "7725019446cf4bb8bf33dcaea89abcf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37eb7e37e27e4c8db99e00f032a71e04" is_pre_election: true
I20260928 15:45:39.454613 23869 leader_election.cc:302] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4341f72d1c5d4960b55529e71fea0168, 7725019446cf4bb8bf33dcaea89abcf6; no voters:
I20260928 15:45:39.454828 23801 raft_consensus.cc:2454] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7725019446cf4bb8bf33dcaea89abcf6 in term 0.
I20260928 15:45:39.455070 24142 raft_consensus.cc:2781] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:45:39.455206 24142 raft_consensus.cc:482] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:45:39.455295 24142 raft_consensus.cc:3037] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:39.458570 24142 raft_consensus.cc:504] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.459256 24142 leader_election.cc:288] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [CANDIDATE]: Term 1 election: Requested vote from peers 37eb7e37e27e4c8db99e00f032a71e04 (127.13.5.65:43905), 4341f72d1c5d4960b55529e71fea0168 (127.13.5.67:43885)
I20260928 15:45:39.459805 23801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea77a82f051346459c099b915d7b86e2" candidate_uuid: "7725019446cf4bb8bf33dcaea89abcf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37eb7e37e27e4c8db99e00f032a71e04"
I20260928 15:45:39.460067 23801 raft_consensus.cc:3037] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:39.460100 24072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea77a82f051346459c099b915d7b86e2" candidate_uuid: "7725019446cf4bb8bf33dcaea89abcf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4341f72d1c5d4960b55529e71fea0168"
I20260928 15:45:39.460347 24072 raft_consensus.cc:3037] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:45:39.462796 23801 raft_consensus.cc:2454] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7725019446cf4bb8bf33dcaea89abcf6 in term 1.
I20260928 15:45:39.462822 24072 raft_consensus.cc:2454] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7725019446cf4bb8bf33dcaea89abcf6 in term 1.
I20260928 15:45:39.463191 23869 leader_election.cc:302] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4341f72d1c5d4960b55529e71fea0168, 7725019446cf4bb8bf33dcaea89abcf6; no voters:
I20260928 15:45:39.463469 24142 raft_consensus.cc:2781] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:45:39.463857 24142 raft_consensus.cc:686] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 1 LEADER]: Becoming Leader. State: Replica: 7725019446cf4bb8bf33dcaea89abcf6, State: Running, Role: LEADER
I20260928 15:45:39.464489 24142 consensus_queue.cc:237] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } }
I20260928 15:45:39.468958 23657 catalog_manager.cc:5495] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7725019446cf4bb8bf33dcaea89abcf6 (127.13.5.66). New cstate: current_term: 1 leader_uuid: "7725019446cf4bb8bf33dcaea89abcf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7725019446cf4bb8bf33dcaea89abcf6" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 32821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 } health_report { overall_health: UNKNOWN } } }
I20260928 15:45:39.508040 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.502134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.13.5.66:32821)
0928 15:45:39.502355 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.502363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.502444 (+ 81us) server_negotiation.cc:407] Connection header received
0928 15:45:39.502769 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.502773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.502863 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.503094 (+ 231us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:45:39.503481 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.504234 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.505707 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.506171 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.506315 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.506817 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:45:39.507342 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:45:39.507507 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.507591 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.507677 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:45:39.516891 24072 raft_consensus.cc:1261] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 1 FOLLOWER]: Refusing update from remote peer 7725019446cf4bb8bf33dcaea89abcf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:39.516891 23801 raft_consensus.cc:1261] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 1 FOLLOWER]: Refusing update from remote peer 7725019446cf4bb8bf33dcaea89abcf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:45:39.517923 24146 consensus_queue.cc:1035] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4341f72d1c5d4960b55529e71fea0168" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 43885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:39.518348 24142 consensus_queue.cc:1035] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37eb7e37e27e4c8db99e00f032a71e04" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:45:39.530376 24149 mvcc.cc:205] Tried to move back new op lower bound from 7334339950650576896 to 7334339950450450432. Current Snapshot: MvccSnapshot[applied={T|T < 7334339950650576896}]
I20260928 15:45:39.544514 23657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:39.544792 23657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:45:39.548480 23657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bc1af40ced14a0084a1a7843f8babc9: Sending DeleteTablet for 3 replicas of tablet ea77a82f051346459c099b915d7b86e2
I20260928 15:45:39.549705 23917 tablet_service.cc:1514] Processing DeleteTablet for tablet ea77a82f051346459c099b915d7b86e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:39 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:40568
I20260928 15:45:39.550063 23781 tablet_service.cc:1514] Processing DeleteTablet for tablet ea77a82f051346459c099b915d7b86e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:39 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:60378
I20260928 15:45:39.550498 24052 tablet_service.cc:1514] Processing DeleteTablet for tablet ea77a82f051346459c099b915d7b86e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:45:39 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:54652
I20260928 15:45:39.551357 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 23718
I20260928 15:45:39.566903 24161 tablet_replica.cc:330] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: stopping tablet replica
I20260928 15:45:39.569082 24160 tablet_replica.cc:330] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: stopping tablet replica
I20260928 15:45:39.569422 24161 raft_consensus.cc:2229] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:39.570114 24161 raft_consensus.cc:2258] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:39.569543 24160 raft_consensus.cc:2229] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:45:39.570851 24159 tablet_replica.cc:330] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: stopping tablet replica
I20260928 15:45:39.571026 24160 raft_consensus.cc:2258] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:39.571636 24159 raft_consensus.cc:2229] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:45:39.573777 24159 raft_consensus.cc:2258] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:45:39.573855 24161 ts_tablet_manager.cc:1905] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:39.573979 24160 ts_tablet_manager.cc:1905] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:45:39.576048 24159 ts_tablet_manager.cc:1905] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:45:39 dist-test-slave-j968 krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:45:39.581588 24161 ts_tablet_manager.cc:1918] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:39.581810 24161 log.cc:1196] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/ea77a82f051346459c099b915d7b86e2
I20260928 15:45:39.581830 24160 ts_tablet_manager.cc:1918] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:39.582003 24160 log.cc:1196] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/ea77a82f051346459c099b915d7b86e2
I20260928 15:45:39.582470 24161 ts_tablet_manager.cc:1939] T ea77a82f051346459c099b915d7b86e2 P 4341f72d1c5d4960b55529e71fea0168: Deleting consensus metadata
I20260928 15:45:39.582683 24160 ts_tablet_manager.cc:1939] T ea77a82f051346459c099b915d7b86e2 P 37eb7e37e27e4c8db99e00f032a71e04: Deleting consensus metadata
I20260928 15:45:39.584559 23641 catalog_manager.cc:4841] TS 4341f72d1c5d4960b55529e71fea0168 (127.13.5.67:43885): tablet ea77a82f051346459c099b915d7b86e2 (table test-table [id=4927541c52c746cb98bd7acfd80e709a]) successfully deleted
I20260928 15:45:39.585176 23641 catalog_manager.cc:4841] TS 37eb7e37e27e4c8db99e00f032a71e04 (127.13.5.65:43905): tablet ea77a82f051346459c099b915d7b86e2 (table test-table [id=4927541c52c746cb98bd7acfd80e709a]) successfully deleted
I20260928 15:45:39.589568 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.552961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.13.5.65:43905)
0928 15:45:39.553180 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.553189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.563415 (+ 10226us) server_negotiation.cc:407] Connection header received
0928 15:45:39.564009 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.564017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.564168 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.564257 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:39.567231 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.568874 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.571418 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.572149 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.572253 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.582168 (+ 9915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:39.582188 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:39.582216 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:39.582278 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:39.586293 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:39.587082 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:39.587091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:39.587095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:39.587243 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:39.587657 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:39.587663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:39.587667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:39.588224 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:39.588540 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.588848 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.589075 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:45:39.590215 23841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:39.591063 24159 ts_tablet_manager.cc:1918] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:45:39.591256 24159 log.cc:1196] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/ea77a82f051346459c099b915d7b86e2
I20260928 15:45:39.591812 24159 ts_tablet_manager.cc:1939] T ea77a82f051346459c099b915d7b86e2 P 7725019446cf4bb8bf33dcaea89abcf6: Deleting consensus metadata
I20260928 15:45:39.593719 23642 catalog_manager.cc:4841] TS 7725019446cf4bb8bf33dcaea89abcf6 (127.13.5.66:32821): tablet ea77a82f051346459c099b915d7b86e2 (table test-table [id=4927541c52c746cb98bd7acfd80e709a]) successfully deleted
I20260928 15:45:39.683382 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 23718
I20260928 15:45:39.703320 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 23852
Sep 28 15:45:39 dist-test-slave-j968 krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:45:39.721843 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.704152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.13.5.66:32821)
0928 15:45:39.704335 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.704342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.704509 (+ 167us) server_negotiation.cc:407] Connection header received
0928 15:45:39.704664 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.704669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.704759 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.704873 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:39.706381 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.707629 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.709045 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.709756 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.709891 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.714193 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:39.714213 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:39.714237 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:39.714308 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:39.718349 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:39.719187 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:39.719197 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:39.719201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:39.719372 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:39.719835 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:39.719841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:39.719844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:39.720395 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:39.720738 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.721119 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.721401 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 15:45:39.722690 23978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:39.844244 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 23852
I20260928 15:45:39.863481 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 23988
Sep 28 15:45:39 dist-test-slave-j968 krb5kdc[23611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:45:39.878239 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:45:39.864357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.13.5.67:43885)
0928 15:45:39.864527 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:45:39.864531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:45:39.864549 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:45:39.864675 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:45:39.864679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:45:39.864744 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:45:39.864823 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:45:39.866115 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.866820 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.868170 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:45:39.868553 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:45:39.868647 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:45:39.872837 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:45:39.872851 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:45:39.872868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:45:39.872907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:45:39.875804 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:39.876468 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:39.876476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:39.876479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:39.876635 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:45:39.876983 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:45:39.876986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:45:39.876988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:45:39.877332 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:45:39.877583 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 15:45:39.877809 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 15:45:39.877972 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:45:39.878922 24112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:39.967581 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 23988
I20260928 15:45:39.985127 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 23624
I20260928 15:45:39.986418 23687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:45:40.098346 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 23624
2026-09-28T15:45:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4625 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:45:40 dist-test-slave-j968 krb5kdc[24170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:45:40 dist-test-slave-j968 krb5kdc[24170](info): set up 2 sockets
Sep 28 15:45:40 dist-test-slave-j968 krb5kdc[24170](info): commencing operation
krb5kdc: starting...
W20260928 15:45:41.279254 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:45:41 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:45:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.5.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.5.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/rangeradmin_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/rangeradmin_127.13.5.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.5.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.5.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/rangerlookup_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/rangerlookup_127.13.5.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.5.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.5.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/HTTP_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/HTTP_127.13.5.84@KRBTEST.COM.keytab.
I20260928 15:45:41.631385 13333 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Pa7uo/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/postgres -l logfile start
2026-09-28 15:45:44.392 UTC [24200] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 15:45:44.392 UTC [24200] LOG: listening on IPv4 address "127.13.5.84", port 46301
2026-09-28 15:45:44.396 UTC [24200] LOG: listening on Unix socket "/tmp/.s.PGSQL.46301"
2026-09-28 15:45:44.420 UTC [24203] LOG: database system was shut down at 2026-09-28 15:45:42 UTC
2026-09-28 15:45:44.424 UTC [24200] LOG: database system is ready to accept connections
I20260928 15:45:45.427003 13333 mini_postgres.cc:96] Postgres bound to 46301
2026-09-28 15:45:45.445 UTC [24211] FATAL: database "slave" does not exist
127.13.5.84:46301 - accepting connections
I20260928 15:45:45.445956 13333 mini_ranger.cc:161] Starting Ranger...
I20260928 15:45:45.473017 13333 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 15:45:46.043682 13333 mini_ranger.cc:87] Created ranger Postgres database
I20260928 15:45:46.043813 13333 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-admin
2026-09-28 15:45:46,332 [I] DB FLAVOR :POSTGRES
2026-09-28 15:45:46,333 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:45:46,333 [I] Checking connection..
2026-09-28 15:45:46,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 15:45:46,580 [I] Checking connection passed.
2026-09-28 15:45:46,581 [I] --------- Verifying version history table ---------
2026-09-28 15:45:46,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:45:46,821 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:45:46,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:45:47,076 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:45:47,076 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 15:45:47,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 15:45:47.327 UTC [24275] WARNING: there is no transaction in progress
2026-09-28 15:45:47,334 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 15:45:47,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:45:47,585 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 15:45:47,585 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:45:47,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 15:45:47,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 15:45:48,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j968.c.gcp-upstream.internal','N') ;"
2026-09-28 15:45:48,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:45:48,583 [I] Table x_portal_user does not exist in database ranger
2026-09-28 15:45:48,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:45:48,828 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 15:45:48,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:45:49,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:45:49,296 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 15:45:49,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 15:45:50.307 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.323 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.342 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.358 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.376 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.395 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.452 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.459 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.474 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.485 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.495 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.508 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.518 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.524 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.531 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:50.538 UTC [24384] WARNING: there is no transaction in progress
2026-09-28 15:45:51,223 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 15:45:51,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 15:45:51,460 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 15:45:51,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:45:51,722 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 15:45:51,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:45:51,977 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 15:45:51,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:45:52,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:45:52,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j968.c.gcp-upstream.internal';"
2026-09-28 15:45:52,730 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 15:45:52,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 15:45:52,981 [I] DB_PATCHES have already been applied
I20260928 15:45:52.987519 13333 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Pa7uo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 15:45:52.987630 13333 mini_ranger.cc:193] Using host: 127.13.5.84
I20260928 15:45:53.003540 13333 mini_ranger.cc:227] Ranger admin URL: http://127.13.5.84:53805
Sep 28, 2026 3:45:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:45:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:45:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:45:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:45:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:45:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:45:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 15:45:54.184484 13333 mini_ranger.cc:160] Time spent starting Ranger: real 8.739s user 0.011s sys 0.032s
Sep 28, 2026 3:46:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:46:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:46:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:46:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:46:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:46:15.513042 13333 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.5.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.5.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/rangerkms_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/rangerkms_127.13.5.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.5.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.5.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.5.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/HTTP_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.5.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/HTTP_127.13.5.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:46:15 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610375, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:46:15.702059 13333 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:46:15.724079 13333 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:46:16.724759 13333 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:46:16.724959 13333 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms
I20260928 15:46:16.725106 13333 mini_ranger_kms.cc:227] Using postgres at 127.13.5.84:46301
2026-09-28 15:46:17,199 [I] DB FLAVOR :POSTGRES
2026-09-28 15:46:17,200 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:46:17,200 [I] Checking connection
2026-09-28 15:46:17,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:46:17,460 [I] connection success
2026-09-28 15:46:17,461 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:46:17,461 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:46:17,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:46:17,732 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:46:17,732 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:46:17,732 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:46:17,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:46301/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:46:17,988 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:46:18.434310 13333 mini_ranger_kms.cc:310] Created kms service
I20260928 15:46:18.635836 13333 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:46:18.681567 13333 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:46:18 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610375, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:46:19.120046 13333 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:46:19.120391 13333 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:46:19.120463 13333 mini_ranger_kms.cc:240] Using host: 127.13.5.84
I20260928 15:46:19.138417 13333 mini_ranger_kms.cc:276] Ranger KMS PID: 24565
I20260928 15:46:19.138675 13333 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.5.84:40925
Sep 28, 2026 3:46:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:46:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:46:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:46:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:46:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:46:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 15:46:21.476784 13333 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.775s user 0.117s sys 0.371s
I20260928 15:46:21.477063 13333 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:46:21.477291 13333 mini_ranger_kms.cc:401] 127.13.5.84:40925/kms/v1/keys
Sep 28 15:46:21 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610381, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.5.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 15:46:22 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610381, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.5.84@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
Sep 28 15:46:22 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610381, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.5.84@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
Sep 28 15:46:24 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:46:24.796011 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36593
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:36593
--ranger_config_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40925/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:46:25.261021 24616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:46:25.261339 24616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:46:25.261581 24616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:46:25.269618 24616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:46:25.269837 24616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:46:25.269994 24616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:46:25.270120 24616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:46:25.270277 24616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:46:25.282131 24616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.5.84:40925/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:36593
--ranger_config_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:46:25.283593 24616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:46:25.285200 24616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:46:25.303056 24621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:46:25.303260 24622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:46:25.302944 24624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:46:25.304864 24616 server_base.cc:1034] running on GCE node
I20260928 15:46:25.305740 24616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:46:25.307396 24616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:46:25.308631 24616 hybrid_clock.cc:648] HybridClock initialized: now 1790610385308572 us; error 52 us; skew 500 ppm
I20260928 15:46:25.309000 24616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:46:25 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610385, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:46:25.313042 24616 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:46:25.315191 24616 webserver.cc:460] Webserver started at http://127.13.5.126:42901/ using document root <none> and password file <none>
I20260928 15:46:25.315934 24616 fs_manager.cc:359] Metadata directory not provided
I20260928 15:46:25.316088 24616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:46:25.316461 24616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:46:25 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610385, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:46:25.379539 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "1a830edfcf744023acb48767d86eae12"
format_stamp: "Formatted at 2026-09-28 15:46:25 on dist-test-slave-j968"
server_key: "d43898178bebbb45f002eebcbc57b491"
server_key_iv: "55dc505574eea05012791dd8b06140cb"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:25.380545 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "1a830edfcf744023acb48767d86eae12"
format_stamp: "Formatted at 2026-09-28 15:46:25 on dist-test-slave-j968"
server_key: "d43898178bebbb45f002eebcbc57b491"
server_key_iv: "55dc505574eea05012791dd8b06140cb"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:25.386924 24616 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 15:46:25.404945 24631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:25.406731 24616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 15:46:25.406934 24616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "1a830edfcf744023acb48767d86eae12"
format_stamp: "Formatted at 2026-09-28 15:46:25 on dist-test-slave-j968"
server_key: "d43898178bebbb45f002eebcbc57b491"
server_key_iv: "55dc505574eea05012791dd8b06140cb"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:25.407157 24616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:46:25.432344 24616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:46:25.437108 24616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:46:25.437443 24616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:46:25.469264 24616 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:36593
I20260928 15:46:25.469272 24682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:36593 every 8 connection(s)
I20260928 15:46:25.470956 24616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:46:25.474128 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 24616
I20260928 15:46:25.474351 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:46:25.475257 13333 external_mini_cluster.cc:1428] Setting key fe12b23da1c1916fda28c496967d9ebb
I20260928 15:46:25.477541 24683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:46:25.498828 24683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12: Bootstrap starting.
I20260928 15:46:25.503671 24683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12: Neither blocks nor log segments found. Creating new log.
I20260928 15:46:25.505246 24683 log.cc:824] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:46:25 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:46:25.508883 24683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12: No bootstrap required, opened a new log
I20260928 15:46:25.520582 24683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a830edfcf744023acb48767d86eae12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36593 } }
I20260928 15:46:25.521111 24683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:46:25.521131 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:25.477641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.13.5.126:36593)
0928 15:46:25.488768 (+ 11127us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:25.488783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:25.492225 (+ 3442us) server_negotiation.cc:407] Connection header received
0928 15:46:25.496303 (+ 4078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:25.496338 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:25.497025 (+ 687us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:25.497539 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:46:25.499143 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:25.501708 (+ 2565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:25.503514 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:25.504702 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:25.504888 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:25.510058 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:46:25.510069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:46:25.510091 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:46:25.510137 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:46:25.515211 (+ 5074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:25.516379 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:25.516389 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:25.516398 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:25.516597 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:25.517145 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:25.517150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:25.517153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:25.517915 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:46:25.518306 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:25.518807 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 15:46:25.519433 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10806,"thread_start_us":10439,"threads_started":1}
I20260928 15:46:25.521272 24683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a830edfcf744023acb48767d86eae12, State: Initialized, Role: FOLLOWER
I20260928 15:46:25.522061 24683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a830edfcf744023acb48767d86eae12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36593 } }
I20260928 15:46:25.522369 24683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:46:25.522670 24683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:46:25.522877 24683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:46:25.525653 24683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a830edfcf744023acb48767d86eae12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36593 } }
I20260928 15:46:25.526228 24683 leader_election.cc:302] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a830edfcf744023acb48767d86eae12; no voters:
I20260928 15:46:25.526971 24683 leader_election.cc:288] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:46:25.527259 24688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:46:25.530270 24683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:46:25.536365 24688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [term 1 LEADER]: Becoming Leader. State: Replica: 1a830edfcf744023acb48767d86eae12, State: Running, Role: LEADER
I20260928 15:46:25.537672 24688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a830edfcf744023acb48767d86eae12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36593 } }
I20260928 15:46:25.542197 24689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a830edfcf744023acb48767d86eae12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a830edfcf744023acb48767d86eae12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36593 } } }
I20260928 15:46:25.542991 24689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [sys.catalog]: This master's current role is: LEADER
I20260928 15:46:25.543429 24690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a830edfcf744023acb48767d86eae12. Latest consensus state: current_term: 1 leader_uuid: "1a830edfcf744023acb48767d86eae12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a830edfcf744023acb48767d86eae12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36593 } } }
I20260928 15:46:25.543792 24690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12 [sys.catalog]: This master's current role is: LEADER
I20260928 15:46:25.557562 24698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:46:25.573076 24698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:46:25.573959 24683 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:46:25.612484 24698 catalog_manager.cc:1347] Generated new cluster ID: 45f85625cea749fca4b71551da9f026d
I20260928 15:46:25.612644 24698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:46:25.642268 24698 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:46:25.644989 24698 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:46:25.660055 24698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12: Generated new TSK 0
I20260928 15:46:25.661564 24698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:46:26.494352 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40925/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36593
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:46:26 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610386, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 15:46:27.003446 24730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:46:27.003988 24730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:46:27.004207 24730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:46:27.016161 24730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:46:27.016347 24730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:46:27.016606 24730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:46:27.027272 24730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.5.84:40925/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:46:27.028666 24730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:46:27.030037 24730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:46:27.050770 24741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:46:27.060312 24742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:46:27.071055 24730 server_base.cc:1034] running on GCE node
W20260928 15:46:27.076344 24744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:46:27.077551 24730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:46:27.079368 24730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:46:27.080742 24730 hybrid_clock.cc:648] HybridClock initialized: now 1790610387080628 us; error 98 us; skew 500 ppm
I20260928 15:46:27.081174 24730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:46:27 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610387, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:46:27.086911 24730 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:46:27.099946 24730 webserver.cc:460] Webserver started at http://127.13.5.65:36693/ using document root <none> and password file <none>
I20260928 15:46:27.100982 24730 fs_manager.cc:359] Metadata directory not provided
I20260928 15:46:27.101136 24730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:46:27.101671 24730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:46:27 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610387, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:46:27.124838 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "7971a81d28d8497d9803eed60058e974"
format_stamp: "Formatted at 2026-09-28 15:46:27 on dist-test-slave-j968"
server_key: "5fc691e10e11741c79737b135e5f2781"
server_key_iv: "6f0e70b6d451a688ce863b106567a62e"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:27.126297 24730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "7971a81d28d8497d9803eed60058e974"
format_stamp: "Formatted at 2026-09-28 15:46:27 on dist-test-slave-j968"
server_key: "5fc691e10e11741c79737b135e5f2781"
server_key_iv: "6f0e70b6d451a688ce863b106567a62e"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:27.153156 24730 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.018s sys 0.009s
I20260928 15:46:27.187942 24751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:27.194344 24730 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.010s sys 0.004s
I20260928 15:46:27.194689 24730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "7971a81d28d8497d9803eed60058e974"
format_stamp: "Formatted at 2026-09-28 15:46:27 on dist-test-slave-j968"
server_key: "5fc691e10e11741c79737b135e5f2781"
server_key_iv: "6f0e70b6d451a688ce863b106567a62e"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:27.195122 24730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:46:27.258410 24730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:46:27.265373 24730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:46:27.266180 24730 kserver.cc:163] Server-wide thread pool size limit: 3276
Sep 28 15:46:27 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610386, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:46:27.274710 24730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:46:27.283352 24730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:46:27.283550 24730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:46:27.283810 24730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:46:27.284145 24730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:27.576498 24730 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:45455
I20260928 15:46:27.576538 24864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:45455 every 8 connection(s)
I20260928 15:46:27.578544 24730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:46:27.579308 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 24730
I20260928 15:46:27.579622 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:46:27.580631 13333 external_mini_cluster.cc:1428] Setting key 75ecbbcb243b5e365359513974750dab
Sep 28 15:46:27 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610387, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
I20260928 15:46:27.629093 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:27.586358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:54705 (local address 127.13.5.126:36593)
0928 15:46:27.592954 (+ 6596us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:27.592962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:27.592995 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:46:27.594915 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:27.594937 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:27.595126 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:27.595228 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:46:27.597901 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:27.599238 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:27.602619 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:27.603678 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:27.603799 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:27.618694 (+ 14895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:46:27.618702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:46:27.618707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:46:27.618766 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:46:27.623161 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:27.624722 (+ 1561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:27.624730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:27.624734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:27.625001 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:27.625978 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:27.625984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:27.625987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:27.626666 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:46:27.626817 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:27.628126 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0928 15:46:27.628427 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6423,"thread_start_us":6249,"threads_started":1}
I20260928 15:46:27.632189 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:27.585405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36593 (local address 127.13.5.65:54705)
0928 15:46:27.588204 (+ 2799us) negotiation.cc:107] Waiting for socket to connect
0928 15:46:27.588277 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 15:46:27.594163 (+ 5886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:46:27.595848 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:46:27.595861 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:46:27.596693 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:46:27.597611 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:27.597648 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:27.599470 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:27.599475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:27.602377 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:27.602440 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:27.604248 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:27.604256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:27.604613 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:27.615164 (+ 10551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:46:27.615204 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:46:27.618345 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:46:27.623452 (+ 5107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:46:27.623467 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:46:27.623494 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:46:27.624519 (+ 1025us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:46:27.625220 (+ 701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:46:27.625226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:46:27.625230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:46:27.625786 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:46:27.627002 (+ 1216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:46:27.627013 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:46:27.627516 (+ 503us) client_negotiation.cc:769] Sending connection context
0928 15:46:27.629664 (+ 2148us) client_negotiation.cc:240] Negotiation successful
0928 15:46:27.630395 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2392,"thread_start_us":2102,"threads_started":1}
I20260928 15:46:27.634234 24865 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36593
I20260928 15:46:27.634691 24865 heartbeater.cc:464] Registering TS with master...
I20260928 15:46:27.636147 24865 heartbeater.cc:511] Master 127.13.5.126:36593 requested a full tablet report, sending...
I20260928 15:46:27.639213 24648 ts_manager.cc:194] Registered new tserver with Master: 7971a81d28d8497d9803eed60058e974 (127.13.5.65:45455)
I20260928 15:46:27.641379 24648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:54705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:46:27.668119 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40925/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36593
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:46:27 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610386, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
W20260928 15:46:28.240278 24871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:46:28.240644 24871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:46:28.240756 24871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:46:28.249074 24871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:46:28.249178 24871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:46:28.249291 24871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:46:28.260011 24871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.5.84:40925/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:46:28.261289 24871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:46:28.262656 24871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:46:28.276743 24886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:46:28.276780 24887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:46:28.276837 24889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:46:28.277747 24871 server_base.cc:1034] running on GCE node
I20260928 15:46:28.278463 24871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:46:28.279508 24871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:46:28.280707 24871 hybrid_clock.cc:648] HybridClock initialized: now 1790610388280643 us; error 58 us; skew 500 ppm
I20260928 15:46:28.281008 24871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:46:28 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610388, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:46:28.284976 24871 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:46:28.286473 24871 webserver.cc:460] Webserver started at http://127.13.5.66:41823/ using document root <none> and password file <none>
I20260928 15:46:28.287111 24871 fs_manager.cc:359] Metadata directory not provided
I20260928 15:46:28.287211 24871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:46:28.287554 24871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:46:28 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610388, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:46:28.305027 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "a273d15fdfc041079070563dc1245564"
format_stamp: "Formatted at 2026-09-28 15:46:28 on dist-test-slave-j968"
server_key: "ddc84382c9feed0eb0c83cfed3e19089"
server_key_iv: "0633e5ba35328a61a54a4f444648deb8"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:28.306032 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "a273d15fdfc041079070563dc1245564"
format_stamp: "Formatted at 2026-09-28 15:46:28 on dist-test-slave-j968"
server_key: "ddc84382c9feed0eb0c83cfed3e19089"
server_key_iv: "0633e5ba35328a61a54a4f444648deb8"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:28.312161 24871 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 15:46:28.330888 24896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:28.332919 24871 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 15:46:28.333114 24871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "a273d15fdfc041079070563dc1245564"
format_stamp: "Formatted at 2026-09-28 15:46:28 on dist-test-slave-j968"
server_key: "ddc84382c9feed0eb0c83cfed3e19089"
server_key_iv: "0633e5ba35328a61a54a4f444648deb8"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:28.333480 24871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:46:28.364598 24871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:46:28.369976 24871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:46:28.370329 24871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:46:28.371726 24871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:46:28.373941 24871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:46:28.374046 24871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:28.374192 24871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:46:28.374250 24871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:28.431674 24871 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:44929
I20260928 15:46:28.431929 25008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:44929 every 8 connection(s)
I20260928 15:46:28.433393 24871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:46:28.438328 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 24871
I20260928 15:46:28.438540 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:46:28.439060 13333 external_mini_cluster.cc:1428] Setting key f7e269a8e3d4c7249ae216d4f9cbbaa3
Sep 28 15:46:28 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610388, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:46:28.466243 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:28.437219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:54211 (local address 127.13.5.126:36593)
0928 15:46:28.437831 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:28.437835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:28.442402 (+ 4567us) server_negotiation.cc:407] Connection header received
0928 15:46:28.445386 (+ 2984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:28.445392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:28.445483 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:28.445607 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:46:28.448443 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:28.449769 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:28.452272 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:28.453181 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:28.453330 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:28.458078 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:46:28.458085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:46:28.458090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:46:28.458138 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:46:28.461832 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:28.463005 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:28.463012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:28.463016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:28.463177 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:28.463775 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:28.463781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:28.463784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:28.464195 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:46:28.464355 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:28.465586 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0928 15:46:28.465851 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":392,"threads_started":1}
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:46:28.467679 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:28.437862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36593 (local address 127.13.5.66:54211)
0928 15:46:28.441189 (+ 3327us) negotiation.cc:107] Waiting for socket to connect
0928 15:46:28.441243 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:46:28.443688 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:46:28.445975 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:46:28.445989 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:46:28.446995 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:46:28.448233 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:28.448263 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:28.449947 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:28.449952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:28.452109 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:28.452125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:28.453445 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:28.453455 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:28.453775 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:28.454642 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:46:28.454677 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:46:28.457776 (+ 3099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:46:28.462047 (+ 4271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:46:28.462059 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:46:28.462081 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:46:28.462834 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:46:28.463310 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:46:28.463317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:46:28.463322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:46:28.463617 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:46:28.464379 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:46:28.464389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:46:28.465060 (+ 671us) client_negotiation.cc:769] Sending connection context
0928 15:46:28.465432 (+ 372us) client_negotiation.cc:240] Negotiation successful
0928 15:46:28.466066 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3061,"thread_start_us":1978,"threads_started":1}
I20260928 15:46:28.469332 25009 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36593
I20260928 15:46:28.469859 25009 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
I20260928 15:46:28.471304 25009 heartbeater.cc:511] Master 127.13.5.126:36593 requested a full tablet report, sending...
I20260928 15:46:28.473389 24648 ts_manager.cc:194] Registered new tserver with Master: a273d15fdfc041079070563dc1245564 (127.13.5.66:44929)
I20260928 15:46:28.474445 24648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:54211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:46:28.507920 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40925/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36593
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:46:28.649943 24865 heartbeater.cc:503] Master 127.13.5.126:36593 was elected leader, sending a full tablet report...
W20260928 15:46:28.980692 25015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:46:28.980983 25015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:46:28.981189 25015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:46:28.990341 25015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:46:28.990600 25015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:46:28.990859 25015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:46:29.003363 25015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.5.84:40925/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:36593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:46:29.004832 25015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:46:29.006171 25015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:46:29.020977 25021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:46:29.021322 25022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:46:29.021425 25024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:46:29.023002 25015 server_base.cc:1034] running on GCE node
I20260928 15:46:29.023795 25015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:46:29.025234 25015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:46:29.026585 25015 hybrid_clock.cc:648] HybridClock initialized: now 1790610389026453 us; error 53 us; skew 500 ppm
I20260928 15:46:29.026950 25015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:46:29 dist-test-slave-j968 krb5kdc[24170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610389, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:46:29.030912 25015 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:46:29.032574 25015 webserver.cc:460] Webserver started at http://127.13.5.67:42981/ using document root <none> and password file <none>
I20260928 15:46:29.033217 25015 fs_manager.cc:359] Metadata directory not provided
I20260928 15:46:29.033317 25015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:46:29.033581 25015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:46:29 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610389, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:46:29.050932 25015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "b2135d73b1104cf6876c83bd1f42d959"
format_stamp: "Formatted at 2026-09-28 15:46:29 on dist-test-slave-j968"
server_key: "233eca335849d226a1c3efc312786830"
server_key_iv: "786ab04d3a8d9e25dd0522dfc5baf670"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:29.052110 25015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "b2135d73b1104cf6876c83bd1f42d959"
format_stamp: "Formatted at 2026-09-28 15:46:29 on dist-test-slave-j968"
server_key: "233eca335849d226a1c3efc312786830"
server_key_iv: "786ab04d3a8d9e25dd0522dfc5baf670"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:29.058420 25015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:46:29.077008 25031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:29.078626 25015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:46:29.078826 25015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "b2135d73b1104cf6876c83bd1f42d959"
format_stamp: "Formatted at 2026-09-28 15:46:29 on dist-test-slave-j968"
server_key: "233eca335849d226a1c3efc312786830"
server_key_iv: "786ab04d3a8d9e25dd0522dfc5baf670"
server_key_version: "kuduclusterkey@0"
I20260928 15:46:29.079080 25015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:46:29.108882 25015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:46:29.114262 25015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:46:29.114810 25015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:46:29.116321 25015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:46:29.118001 25015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:46:29.118114 25015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:29.118266 25015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:46:29.118338 25015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:46:29.178315 25015 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:33945
I20260928 15:46:29.178444 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:33945 every 8 connection(s)
I20260928 15:46:29.180194 25015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:46:29.187531 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 25015
I20260928 15:46:29.187731 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:46:29.188385 13333 external_mini_cluster.cc:1428] Setting key 0914e0197263f80c8be9c5e93852421a
Sep 28 15:46:29 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610389, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:46:29.206025 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.183119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:52721 (local address 127.13.5.126:36593)
0928 15:46:29.184235 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.184244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.184700 (+ 456us) server_negotiation.cc:407] Connection header received
0928 15:46:29.186618 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.186624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.186721 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.186839 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:46:29.188747 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.189811 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.193362 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.193933 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.194049 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.198539 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:46:29.198545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:46:29.198548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:46:29.198583 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:46:29.202224 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.203186 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.203194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.203198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.203389 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.203971 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.203977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.203980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.204500 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:46:29.204653 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.205479 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.205695 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":708,"threads_started":1}
I20260928 15:46:29.207007 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.183677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:36593 (local address 127.13.5.67:52721)
0928 15:46:29.184533 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0928 15:46:29.184573 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:46:29.186329 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:46:29.187154 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:46:29.187163 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:46:29.187712 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:46:29.188558 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.188590 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.189982 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.189992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.193159 (+ 3167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.193177 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.194174 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.194180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.194505 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.195325 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:46:29.195352 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:46:29.198236 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:46:29.202385 (+ 4149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:46:29.202394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:46:29.202414 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:46:29.203037 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:46:29.203518 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:46:29.203526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:46:29.203529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:46:29.203772 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:46:29.204637 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:46:29.204643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:46:29.205058 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 15:46:29.205357 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 15:46:29.205779 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":413,"threads_started":1}
I20260928 15:46:29.208622 25144 heartbeater.cc:347] Connected to a master server at 127.13.5.126:36593
I20260928 15:46:29.209044 25144 heartbeater.cc:464] Registering TS with master...
I20260928 15:46:29.210057 25144 heartbeater.cc:511] Master 127.13.5.126:36593 requested a full tablet report, sending...
I20260928 15:46:29.211719 24648 ts_manager.cc:194] Registered new tserver with Master: b2135d73b1104cf6876c83bd1f42d959 (127.13.5.67:33945)
I20260928 15:46:29.212883 24648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:52721
I20260928 15:46:29.217571 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:46:29.239041 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.223505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.13.5.126:36593)
0928 15:46:29.223683 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.223688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.224363 (+ 675us) server_negotiation.cc:407] Connection header received
0928 15:46:29.224591 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.224603 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.224763 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.224870 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:46:29.226716 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.227730 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.229298 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.229987 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.230134 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.233440 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:46:29.233491 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:46:29.233494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:46:29.233529 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:46:29.236342 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.237017 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.237021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.237023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.237147 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.237593 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.237597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.237599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.237899 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:46:29.238104 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.238504 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.238689 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:46:29.244258 24648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:46:29.253679 24648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:46:29.280040 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.269063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:44929 (local address 127.0.0.1:54198)
0928 15:46:29.270001 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0928 15:46:29.270045 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:46:29.270209 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:46:29.271402 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:46:29.271412 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:46:29.271449 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:46:29.271968 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.271983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.274187 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.274193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.276796 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.276812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.278810 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.278817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.279020 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.279096 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 15:46:29.279205 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:46:29.279385 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":1,"thread_start_us":539,"threads_started":1}
I20260928 15:46:29.280198 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.269598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.13.5.67:33945)
0928 15:46:29.270681 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.270689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.271236 (+ 547us) server_negotiation.cc:407] Connection header received
0928 15:46:29.271549 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.271560 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.271806 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.272019 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:46:29.273004 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.274530 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.277107 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.278580 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.278704 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.278949 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.279543 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.279707 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":781,"threads_started":1}
I20260928 15:46:29.281306 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.269163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:33945 (local address 127.0.0.1:52486)
0928 15:46:29.271125 (+ 1962us) negotiation.cc:107] Waiting for socket to connect
0928 15:46:29.271144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:46:29.271345 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:46:29.272338 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:46:29.272342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:46:29.272366 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:46:29.272786 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.272798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.274704 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.274709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.276955 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.276966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.279152 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.279157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.279325 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.279354 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:46:29.280850 (+ 1496us) client_negotiation.cc:240] Negotiation successful
0928 15:46:29.280979 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1844,"thread_start_us":2185,"threads_started":1}
I20260928 15:46:29.281780 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.270202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.13.5.66:44929)
0928 15:46:29.270727 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.270735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.270767 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:46:29.270889 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.270899 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.271176 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.272158 (+ 982us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:46:29.272398 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.274041 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.277046 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.278580 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.280937 (+ 2357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.281134 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.281204 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.281408 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":922,"threads_started":1}
I20260928 15:46:29.285501 25079 tablet_service.cc:1467] Processing CreateTablet for tablet 7b58d35b04de454fa0d8be7929d08857 (DEFAULT_TABLE table=test-table [id=cf77d2bd97834a2495a405c54e4af978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:46:29.285665 24944 tablet_service.cc:1467] Processing CreateTablet for tablet 7b58d35b04de454fa0d8be7929d08857 (DEFAULT_TABLE table=test-table [id=cf77d2bd97834a2495a405c54e4af978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:46:29.287036 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b58d35b04de454fa0d8be7929d08857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:46:29.287068 24944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b58d35b04de454fa0d8be7929d08857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:46:29.292580 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.270966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:45455 (local address 127.0.0.1:53670)
0928 15:46:29.280668 (+ 9702us) negotiation.cc:107] Waiting for socket to connect
0928 15:46:29.280683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:46:29.280775 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:46:29.284455 (+ 3680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:46:29.284460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:46:29.284485 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:46:29.284644 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.284655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.286750 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.286755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.289515 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.289549 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.291640 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.291647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.291855 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.291924 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 15:46:29.292018 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:46:29.292166 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9603,"thread_start_us":12061,"threads_started":1}
I20260928 15:46:29.293709 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.271219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.13.5.65:45455)
0928 15:46:29.283811 (+ 12592us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.283823 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.283854 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:46:29.283977 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.283985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.284228 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.284823 (+ 595us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:46:29.285058 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.286589 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.289721 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.291146 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.291298 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.291538 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.293078 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.293278 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12292,"thread_start_us":13072,"threads_started":1}
I20260928 15:46:29.298552 24799 tablet_service.cc:1467] Processing CreateTablet for tablet 7b58d35b04de454fa0d8be7929d08857 (DEFAULT_TABLE table=test-table [id=cf77d2bd97834a2495a405c54e4af978]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:46:29.300668 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b58d35b04de454fa0d8be7929d08857. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:46:29.302249 25163 tablet_bootstrap.cc:492] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Bootstrap starting.
I20260928 15:46:29.304692 25164 tablet_bootstrap.cc:492] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Bootstrap starting.
I20260928 15:46:29.307652 25163 tablet_bootstrap.cc:654] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Neither blocks nor log segments found. Creating new log.
I20260928 15:46:29.308605 25164 tablet_bootstrap.cc:654] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Neither blocks nor log segments found. Creating new log.
I20260928 15:46:29.309422 25163 log.cc:824] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Log is configured to *not* fsync() on all Append() calls
I20260928 15:46:29.309795 25164 log.cc:824] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Log is configured to *not* fsync() on all Append() calls
I20260928 15:46:29.315177 25164 tablet_bootstrap.cc:492] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: No bootstrap required, opened a new log
I20260928 15:46:29.315469 25164 ts_tablet_manager.cc:1397] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260928 15:46:29.315596 25163 tablet_bootstrap.cc:492] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: No bootstrap required, opened a new log
I20260928 15:46:29.315991 25163 ts_tablet_manager.cc:1397] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260928 15:46:29.319479 25168 tablet_bootstrap.cc:492] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Bootstrap starting.
I20260928 15:46:29.325655 25168 tablet_bootstrap.cc:654] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Neither blocks nor log segments found. Creating new log.
I20260928 15:46:29.327144 25168 log.cc:824] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Log is configured to *not* fsync() on all Append() calls
I20260928 15:46:29.327456 25164 raft_consensus.cc:348] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.327987 25164 raft_consensus.cc:374] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:46:29.328095 25164 raft_consensus.cc:729] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2135d73b1104cf6876c83bd1f42d959, State: Initialized, Role: FOLLOWER
I20260928 15:46:29.328817 25164 consensus_queue.cc:260] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.329571 25163 raft_consensus.cc:348] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.330080 25163 raft_consensus.cc:374] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:46:29.330216 25163 raft_consensus.cc:729] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a273d15fdfc041079070563dc1245564, State: Initialized, Role: FOLLOWER
I20260928 15:46:29.330232 25144 heartbeater.cc:503] Master 127.13.5.126:36593 was elected leader, sending a full tablet report...
I20260928 15:46:29.331077 25164 ts_tablet_manager.cc:1428] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260928 15:46:29.331122 25163 consensus_queue.cc:260] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.332098 25168 tablet_bootstrap.cc:492] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: No bootstrap required, opened a new log
I20260928 15:46:29.332485 25168 ts_tablet_manager.cc:1397] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260928 15:46:29.333623 25009 heartbeater.cc:503] Master 127.13.5.126:36593 was elected leader, sending a full tablet report...
I20260928 15:46:29.334200 25163 ts_tablet_manager.cc:1428] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260928 15:46:29.342947 25168 raft_consensus.cc:348] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.343369 25168 raft_consensus.cc:374] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:46:29.343477 25168 raft_consensus.cc:729] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7971a81d28d8497d9803eed60058e974, State: Initialized, Role: FOLLOWER
I20260928 15:46:29.344116 25168 consensus_queue.cc:260] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.345956 25168 ts_tablet_manager.cc:1428] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260928 15:46:29.432540 25145 tablet.cc:2369] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:46:29.436472 25010 tablet.cc:2369] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:46:29.507009 25171 raft_consensus.cc:482] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:46:29.507522 25171 raft_consensus.cc:504] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.510331 25171 leader_election.cc:288] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2135d73b1104cf6876c83bd1f42d959 (127.13.5.67:33945), 7971a81d28d8497d9803eed60058e974 (127.13.5.65:45455)
I20260928 15:46:29.521196 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.511323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:57819 (local address 127.13.5.65:45455)
0928 15:46:29.511612 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.511620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.512630 (+ 1010us) server_negotiation.cc:407] Connection header received
0928 15:46:29.512919 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.512924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.513006 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.513115 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:46:29.514364 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.515389 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.518689 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.520006 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.520124 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.520231 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.520808 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.520943 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:46:29.521262 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.510912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:36325 (local address 127.13.5.67:33945)
0928 15:46:29.511142 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.511150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.512794 (+ 1644us) server_negotiation.cc:407] Connection header received
0928 15:46:29.512975 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.512980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.513053 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.513166 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:46:29.514724 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.515704 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.518668 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.519955 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.520160 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.520256 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.520633 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.520786 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:46:29.521253 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.510877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:33945 (local address 127.13.5.66:36325)
0928 15:46:29.512685 (+ 1808us) negotiation.cc:107] Waiting for socket to connect
0928 15:46:29.512702 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:46:29.512853 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:46:29.513182 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:46:29.513185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:46:29.513213 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:46:29.514575 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.514585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.515848 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.515854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.518490 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.518505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.520267 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.520279 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.520451 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.520477 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:46:29.520604 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:46:29.520735 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"mutex_wait_us":155,"spinlock_wait_cycles":5504,"thread_start_us":927,"threads_started":1}
I20260928 15:46:29.522151 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.510487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:45455 (local address 127.13.5.66:57819)
0928 15:46:29.512498 (+ 2011us) negotiation.cc:107] Waiting for socket to connect
0928 15:46:29.512525 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:46:29.512751 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:46:29.513167 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:46:29.513171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:46:29.513199 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:46:29.514176 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.514199 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.515556 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.515560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.518491 (+ 2931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:46:29.518505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.520410 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.520419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:46:29.520627 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.520661 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:46:29.521412 (+ 751us) client_negotiation.cc:240] Negotiation successful
0928 15:46:29.521656 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1734,"mutex_wait_us":154,"thread_start_us":1603,"threads_started":1}
I20260928 15:46:29.522861 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b58d35b04de454fa0d8be7929d08857" candidate_uuid: "a273d15fdfc041079070563dc1245564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2135d73b1104cf6876c83bd1f42d959" is_pre_election: true
I20260928 15:46:29.523198 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b58d35b04de454fa0d8be7929d08857" candidate_uuid: "a273d15fdfc041079070563dc1245564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7971a81d28d8497d9803eed60058e974" is_pre_election: true
I20260928 15:46:29.523360 25099 raft_consensus.cc:2454] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a273d15fdfc041079070563dc1245564 in term 0.
I20260928 15:46:29.523715 24820 raft_consensus.cc:2454] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a273d15fdfc041079070563dc1245564 in term 0.
I20260928 15:46:29.524405 24897 leader_election.cc:302] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7971a81d28d8497d9803eed60058e974, a273d15fdfc041079070563dc1245564, b2135d73b1104cf6876c83bd1f42d959; no voters:
I20260928 15:46:29.525112 25171 raft_consensus.cc:2781] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:46:29.525382 25171 raft_consensus.cc:482] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:46:29.525604 25171 raft_consensus.cc:3037] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:46:29.528108 25171 raft_consensus.cc:504] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.528708 25171 leader_election.cc:288] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [CANDIDATE]: Term 1 election: Requested vote from peers b2135d73b1104cf6876c83bd1f42d959 (127.13.5.67:33945), 7971a81d28d8497d9803eed60058e974 (127.13.5.65:45455)
I20260928 15:46:29.529242 24820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b58d35b04de454fa0d8be7929d08857" candidate_uuid: "a273d15fdfc041079070563dc1245564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7971a81d28d8497d9803eed60058e974"
I20260928 15:46:29.529240 25099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b58d35b04de454fa0d8be7929d08857" candidate_uuid: "a273d15fdfc041079070563dc1245564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2135d73b1104cf6876c83bd1f42d959"
I20260928 15:46:29.529455 25099 raft_consensus.cc:3037] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:46:29.529460 24820 raft_consensus.cc:3037] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:46:29.533231 24820 raft_consensus.cc:2454] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a273d15fdfc041079070563dc1245564 in term 1.
I20260928 15:46:29.533234 25099 raft_consensus.cc:2454] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a273d15fdfc041079070563dc1245564 in term 1.
I20260928 15:46:29.533767 24897 leader_election.cc:302] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7971a81d28d8497d9803eed60058e974, a273d15fdfc041079070563dc1245564, b2135d73b1104cf6876c83bd1f42d959; no voters:
I20260928 15:46:29.534132 25171 raft_consensus.cc:2781] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:46:29.534696 25171 raft_consensus.cc:686] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 1 LEADER]: Becoming Leader. State: Replica: a273d15fdfc041079070563dc1245564, State: Running, Role: LEADER
I20260928 15:46:29.535374 25171 consensus_queue.cc:237] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } }
I20260928 15:46:29.540930 24647 catalog_manager.cc:5495] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 reported cstate change: term changed from 0 to 1, leader changed from <none> to a273d15fdfc041079070563dc1245564 (127.13.5.66). New cstate: current_term: 1 leader_uuid: "a273d15fdfc041079070563dc1245564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a273d15fdfc041079070563dc1245564" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 44929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 } health_report { overall_health: UNKNOWN } } }
I20260928 15:46:29.591981 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.586359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.13.5.66:44929)
0928 15:46:29.586584 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.586588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.586607 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:46:29.586750 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.586753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.586839 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.586964 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:46:29.587320 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.588127 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.589540 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.590022 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.590137 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.590599 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:46:29.591180 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:46:29.591303 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.591461 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.591584 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 15:46:29.596166 24866 tablet.cc:2369] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:46:29.600595 24820 raft_consensus.cc:1261] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 1 FOLLOWER]: Refusing update from remote peer a273d15fdfc041079070563dc1245564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:46:29.600628 25099 raft_consensus.cc:1261] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 1 FOLLOWER]: Refusing update from remote peer a273d15fdfc041079070563dc1245564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:46:29.601620 25175 consensus_queue.cc:1035] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7971a81d28d8497d9803eed60058e974" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:46:29.601944 25171 consensus_queue.cc:1035] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2135d73b1104cf6876c83bd1f42d959" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:46:29.614346 25178 mvcc.cc:205] Tried to move back new op lower bound from 7334340155793711104 to 7334340155541688320. Current Snapshot: MvccSnapshot[applied={T|T < 7334340155793711104}]
I20260928 15:46:29.614594 25179 mvcc.cc:205] Tried to move back new op lower bound from 7334340155793711104 to 7334340155541688320. Current Snapshot: MvccSnapshot[applied={T|T < 7334340155793711104}]
I20260928 15:46:29.633781 24647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:46:29.634191 24647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:46:29.639246 24647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a830edfcf744023acb48767d86eae12: Sending DeleteTablet for 3 replicas of tablet 7b58d35b04de454fa0d8be7929d08857
I20260928 15:46:29.640831 24944 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b58d35b04de454fa0d8be7929d08857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:46:29 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:54198
I20260928 15:46:29.640878 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b58d35b04de454fa0d8be7929d08857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:46:29 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52486
I20260928 15:46:29.642204 24799 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b58d35b04de454fa0d8be7929d08857 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:46:29 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:53670
I20260928 15:46:29.644359 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 24730
I20260928 15:46:29.656164 25188 tablet_replica.cc:330] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: stopping tablet replica
I20260928 15:46:29.656847 25188 raft_consensus.cc:2229] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:46:29.657845 25188 raft_consensus.cc:2258] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:46:29.660669 25189 tablet_replica.cc:330] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: stopping tablet replica
I20260928 15:46:29.660802 25188 ts_tablet_manager.cc:1905] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:46:29.661146 25189 raft_consensus.cc:2229] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:46:29.661698 25189 raft_consensus.cc:2258] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:46:29.666776 25189 ts_tablet_manager.cc:1905] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:46:29.668306 25188 ts_tablet_manager.cc:1918] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:46:29.668485 25188 log.cc:1196] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/7b58d35b04de454fa0d8be7929d08857
I20260928 15:46:29.669044 25188 ts_tablet_manager.cc:1939] T 7b58d35b04de454fa0d8be7929d08857 P a273d15fdfc041079070563dc1245564: Deleting consensus metadata
I20260928 15:46:29.671001 24635 catalog_manager.cc:4841] TS a273d15fdfc041079070563dc1245564 (127.13.5.66:44929): tablet 7b58d35b04de454fa0d8be7929d08857 (table test-table [id=cf77d2bd97834a2495a405c54e4af978]) successfully deleted
I20260928 15:46:29.673542 25190 tablet_replica.cc:330] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: stopping tablet replica
I20260928 15:46:29.674110 25189 ts_tablet_manager.cc:1918] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:46:29.674331 25189 log.cc:1196] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/7b58d35b04de454fa0d8be7929d08857
I20260928 15:46:29.674157 25190 raft_consensus.cc:2229] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:46:29.674896 25189 ts_tablet_manager.cc:1939] T 7b58d35b04de454fa0d8be7929d08857 P b2135d73b1104cf6876c83bd1f42d959: Deleting consensus metadata
I20260928 15:46:29.675181 25190 raft_consensus.cc:2258] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:46:29.677145 24632 catalog_manager.cc:4841] TS b2135d73b1104cf6876c83bd1f42d959 (127.13.5.67:33945): tablet 7b58d35b04de454fa0d8be7929d08857 (table test-table [id=cf77d2bd97834a2495a405c54e4af978]) successfully deleted
Sep 28 15:46:29 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:46:29.680830 25190 ts_tablet_manager.cc:1905] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:46:29.687512 25190 ts_tablet_manager.cc:1918] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:46:29.687687 25190 log.cc:1196] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/7b58d35b04de454fa0d8be7929d08857
I20260928 15:46:29.688293 25190 ts_tablet_manager.cc:1939] T 7b58d35b04de454fa0d8be7929d08857 P 7971a81d28d8497d9803eed60058e974: Deleting consensus metadata
I20260928 15:46:29.689237 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.646241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.13.5.65:45455)
0928 15:46:29.646474 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.646483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.649646 (+ 3163us) server_negotiation.cc:407] Connection header received
0928 15:46:29.649764 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.649773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.649896 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.650070 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:46:29.652387 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.653909 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.664307 (+ 10398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.665042 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.665173 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.680836 (+ 15663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:46:29.680852 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:46:29.680871 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:46:29.680924 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:46:29.685312 (+ 4388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.686335 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.686346 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.686350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.686507 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.686955 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.686960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.686964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.687485 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:46:29.688015 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.688238 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.688725 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:46:29.690415 24860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:46:29.690568 24634 catalog_manager.cc:4841] TS 7971a81d28d8497d9803eed60058e974 (127.13.5.65:45455): tablet 7b58d35b04de454fa0d8be7929d08857 (table test-table [id=cf77d2bd97834a2495a405c54e4af978]) successfully deleted
I20260928 15:46:29.790978 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 24730
I20260928 15:46:29.810174 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 24871
Sep 28 15:46:29 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:46:29.831653 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.811098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.13.5.66:44929)
0928 15:46:29.811309 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.811316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.811340 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:46:29.811493 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.811499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.811597 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.811755 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:46:29.813218 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.814675 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.816098 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.817179 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.817328 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.822143 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:46:29.822166 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:46:29.822191 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:46:29.822334 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:46:29.828086 (+ 5752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.828891 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.828903 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.828907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.829135 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.829605 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.829611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.829616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.830137 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:46:29.830606 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.830829 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.831091 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:46:29.832712 25004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:46:29.951663 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 24871
I20260928 15:46:29.971594 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 25015
Sep 28 15:46:29 dist-test-slave-j968 krb5kdc[24170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:46:29.987116 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:46:29.972580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.13.5.67:33945)
0928 15:46:29.972758 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:46:29.972763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:46:29.972781 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:46:29.972911 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:46:29.972915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:46:29.972978 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:46:29.973059 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:46:29.974398 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.975103 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.976502 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:46:29.976880 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:46:29.976970 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:46:29.981342 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:46:29.981367 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:46:29.981384 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:46:29.981424 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:46:29.984353 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.985031 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.985039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.985041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.985178 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:46:29.985570 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:46:29.985574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:46:29.985576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:46:29.985929 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:46:29.986240 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 15:46:29.986554 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 15:46:29.986779 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:46:29.987762 25139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:46:30.079535 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 25015
I20260928 15:46:30.096856 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 24616
I20260928 15:46:30.098101 24678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:46:30.217581 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 24616
I20260928 15:46:30.240502 13333 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:46:30.575665 13333 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:46:30.575987 13333 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:46:30.921778 13333 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:46:30.922 UTC [24200] LOG: received smart shutdown request
2026-09-28 15:46:30.926 UTC [24200] LOG: background worker "logical replication launcher" (PID 24209) exited with exit code 1
2026-09-28 15:46:30.926 UTC [24204] LOG: shutting down
2026-09-28 15:46:31.117 UTC [24200] LOG: database system is shut down
2026-09-28T15:46:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51145 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:46:31 dist-test-slave-j968 krb5kdc[25204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:46:31 dist-test-slave-j968 krb5kdc[25204](info): set up 2 sockets
Sep 28 15:46:31 dist-test-slave-j968 krb5kdc[25204](info): commencing operation
krb5kdc: starting...
W20260928 15:46:32.374848 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:46:32 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:46:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.5.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.5.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/rangeradmin_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/rangeradmin_127.13.5.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.5.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.5.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/rangerlookup_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/rangerlookup_127.13.5.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.5.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.5.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/HTTP_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/HTTP_127.13.5.84@KRBTEST.COM.keytab.
I20260928 15:46:32.723107 13333 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Pa7uo/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/postgres -l logfile start
2026-09-28 15:46:35.403 UTC [25234] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 15:46:35.403 UTC [25234] LOG: listening on IPv4 address "127.13.5.84", port 47067
2026-09-28 15:46:35.407 UTC [25234] LOG: listening on Unix socket "/tmp/.s.PGSQL.47067"
2026-09-28 15:46:35.432 UTC [25237] LOG: database system was shut down at 2026-09-28 15:46:33 UTC
2026-09-28 15:46:35.436 UTC [25234] LOG: database system is ready to accept connections
I20260928 15:46:36.422315 13333 mini_postgres.cc:96] Postgres bound to 47067
2026-09-28 15:46:36.440 UTC [25245] FATAL: database "slave" does not exist
127.13.5.84:47067 - accepting connections
I20260928 15:46:36.441639 13333 mini_ranger.cc:161] Starting Ranger...
I20260928 15:46:36.469195 13333 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 15:46:37.003454 13333 mini_ranger.cc:87] Created ranger Postgres database
I20260928 15:46:37.003575 13333 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-admin
2026-09-28 15:46:37,303 [I] DB FLAVOR :POSTGRES
2026-09-28 15:46:37,304 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:46:37,304 [I] Checking connection..
2026-09-28 15:46:37,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 15:46:37,558 [I] Checking connection passed.
2026-09-28 15:46:37,558 [I] --------- Verifying version history table ---------
2026-09-28 15:46:37,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:46:37,834 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:46:37,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:46:38,079 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:46:38,079 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 15:46:38,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 15:46:38.332 UTC [25310] WARNING: there is no transaction in progress
2026-09-28 15:46:38,339 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 15:46:38,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:46:38,594 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 15:46:38,594 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:46:38,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 15:46:38,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 15:46:39,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j968.c.gcp-upstream.internal','N') ;"
2026-09-28 15:46:39,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:46:39,584 [I] Table x_portal_user does not exist in database ranger
2026-09-28 15:46:39,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:46:39,848 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 15:46:39,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:46:40,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:46:40,369 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 15:46:40,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 15:46:41.393 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.411 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.429 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.448 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.466 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.485 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.550 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.558 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.573 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.585 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.596 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.611 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.621 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.629 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.637 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:41.645 UTC [25418] WARNING: there is no transaction in progress
2026-09-28 15:46:42,342 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 15:46:42,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 15:46:42,604 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 15:46:42,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:46:42,872 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 15:46:42,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:46:43,109 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 15:46:43,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:46:43,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:46:43,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j968.c.gcp-upstream.internal';"
2026-09-28 15:46:43,863 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 15:46:43,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 15:46:44,121 [I] DB_PATCHES have already been applied
I20260928 15:46:44.127508 13333 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Pa7uo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 15:46:44.127632 13333 mini_ranger.cc:193] Using host: 127.13.5.84
I20260928 15:46:44.145267 13333 mini_ranger.cc:227] Ranger admin URL: http://127.13.5.84:41965
Sep 28, 2026 3:46:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:46:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:46:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:46:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:46:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:46:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:46:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 15:46:45.296036 13333 mini_ranger.cc:160] Time spent starting Ranger: real 8.854s user 0.006s sys 0.038s
Sep 28, 2026 3:47:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:47:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:47:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:47:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:47:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:47:06.810092 13333 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.5.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.5.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/rangerkms_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.5.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/rangerkms_127.13.5.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.5.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.5.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.5.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/HTTP_127.13.5.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.5.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/HTTP_127.13.5.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:47:06 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610426, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:47:06.981734 13333 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:47:07.003453 13333 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:47:07.989559 13333 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:47:07.989689 13333 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms
I20260928 15:47:07.989735 13333 mini_ranger_kms.cc:227] Using postgres at 127.13.5.84:47067
2026-09-28 15:47:08,345 [I] DB FLAVOR :POSTGRES
2026-09-28 15:47:08,345 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:47:08,345 [I] Checking connection
2026-09-28 15:47:08,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:47:08,594 [I] connection success
2026-09-28 15:47:08,594 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:47:08,594 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:47:08,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:47:08,845 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:47:08,846 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:47:08,846 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:47:08,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.5.84:47067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:47:09,133 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:47:09.620083 13333 mini_ranger_kms.cc:310] Created kms service
I20260928 15:47:09.797641 13333 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:47:09.847011 13333 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:47:09 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610426, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:10.242586 13333 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:47:10.242925 13333 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Pa7uo/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Pa7uo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:47:10.242985 13333 mini_ranger_kms.cc:240] Using host: 127.13.5.84
I20260928 15:47:10.262619 13333 mini_ranger_kms.cc:276] Ranger KMS PID: 25600
I20260928 15:47:10.262768 13333 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.5.84:40237
Sep 28, 2026 3:47:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:47:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:47:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:47:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:47:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:47:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 15:47:12.609690 13333 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.628s user 0.050s sys 0.320s
I20260928 15:47:12.609953 13333 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:47:12.610149 13333 mini_ranger_kms.cc:401] 127.13.5.84:40237/kms/v1/keys
Sep 28 15:47:12 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610432, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.5.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 15:47:13 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610432, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.5.84@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
Sep 28 15:47:13 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610432, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.5.84@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
Sep 28 15:47:15 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:47:16.003811 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43467
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43467
--ranger_config_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40237/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:16.350342 25652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:16.350704 25652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:16.350872 25652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:16.358177 25652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:16.358335 25652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:16.358448 25652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:16.358551 25652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:16.358642 25652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:16.369318 25652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.5.84:40237/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43467
--ranger_config_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:16.370755 25652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:16.372437 25652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:16.386530 25657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:16.386868 25652 server_base.cc:1034] running on GCE node
W20260928 15:47:16.386562 25658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:16.386520 25660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:16.388123 25652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:16.389712 25652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:16.390915 25652 hybrid_clock.cc:648] HybridClock initialized: now 1790610436390851 us; error 58 us; skew 500 ppm
I20260928 15:47:16.391305 25652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:16 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610436, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:16.394670 25652 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:47:16.396363 25652 webserver.cc:460] Webserver started at http://127.13.5.126:37687/ using document root <none> and password file <none>
I20260928 15:47:16.397082 25652 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:16.397228 25652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:16.397554 25652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:47:16 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610436, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:16.448239 25652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "d094f2bceb4a4894aaedad432f4d3eb1"
format_stamp: "Formatted at 2026-09-28 15:47:16 on dist-test-slave-j968"
server_key: "c97ba6c72ac38ff6ee02fe78f5fd0e7d"
server_key_iv: "a1bb700d75f9b7fbaf3652d6adf58d06"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:16.449213 25652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "d094f2bceb4a4894aaedad432f4d3eb1"
format_stamp: "Formatted at 2026-09-28 15:47:16 on dist-test-slave-j968"
server_key: "c97ba6c72ac38ff6ee02fe78f5fd0e7d"
server_key_iv: "a1bb700d75f9b7fbaf3652d6adf58d06"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:16.455228 25652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 15:47:16.472210 25667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:16.473946 25652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:47:16.474159 25652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "d094f2bceb4a4894aaedad432f4d3eb1"
format_stamp: "Formatted at 2026-09-28 15:47:16 on dist-test-slave-j968"
server_key: "c97ba6c72ac38ff6ee02fe78f5fd0e7d"
server_key_iv: "a1bb700d75f9b7fbaf3652d6adf58d06"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:16.474406 25652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:16.495591 25652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:16.500088 25652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:16.500449 25652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:16.519541 25652 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43467
I20260928 15:47:16.519591 25719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43467 every 8 connection(s)
I20260928 15:47:16.521056 25652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:47:16.522285 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 25652
I20260928 15:47:16.522508 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:47:16.523168 13333 external_mini_cluster.cc:1428] Setting key e3518ced00e9a5dcc428d452dfd72457
I20260928 15:47:16.525650 25720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:47:16 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:16.538862 25720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Bootstrap starting.
I20260928 15:47:16.542207 25720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:16.543569 25720 log.cc:824] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:16.546912 25720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: No bootstrap required, opened a new log
I20260928 15:47:16.547358 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:16.525239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.13.5.126:43467)
0928 15:47:16.526039 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:16.526055 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:16.526107 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:47:16.528438 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:16.528466 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:16.528994 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:16.529443 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:16.530654 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:16.532781 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:16.534074 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:16.535094 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:16.535289 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:16.539572 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:16.539580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:16.539609 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:16.539650 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:16.542905 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:16.543787 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:16.543803 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:16.543808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:16.544039 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:16.544518 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:16.544524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:16.544534 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:16.545241 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:16.545475 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:16.545930 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 15:47:16.546296 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":383,"threads_started":1}
I20260928 15:47:16.555022 25720 raft_consensus.cc:348] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } }
I20260928 15:47:16.555405 25720 raft_consensus.cc:374] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:16.555529 25720 raft_consensus.cc:729] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d094f2bceb4a4894aaedad432f4d3eb1, State: Initialized, Role: FOLLOWER
I20260928 15:47:16.556149 25720 consensus_queue.cc:260] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } }
I20260928 15:47:16.556373 25720 raft_consensus.cc:388] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:47:16.556528 25720 raft_consensus.cc:482] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:47:16.556701 25720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:16.558807 25720 raft_consensus.cc:504] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } }
I20260928 15:47:16.559309 25720 leader_election.cc:302] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d094f2bceb4a4894aaedad432f4d3eb1; no voters:
I20260928 15:47:16.559957 25720 leader_election.cc:288] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:47:16.560117 25725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:47:16.560793 25725 raft_consensus.cc:686] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 1 LEADER]: Becoming Leader. State: Replica: d094f2bceb4a4894aaedad432f4d3eb1, State: Running, Role: LEADER
I20260928 15:47:16.561448 25725 consensus_queue.cc:237] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } }
I20260928 15:47:16.562297 25720 sys_catalog.cc:564] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:16.564430 25726 sys_catalog.cc:455] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } } }
I20260928 15:47:16.564880 25726 sys_catalog.cc:458] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:16.566710 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d094f2bceb4a4894aaedad432f4d3eb1. Latest consensus state: current_term: 1 leader_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } } }
I20260928 15:47:16.567139 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:16.567840 25732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:47:16.573326 25732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:47:16.573993 25720 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:47:16.592746 25732 catalog_manager.cc:1347] Generated new cluster ID: c0ddf69cfc444b0c825941586777a9a5
I20260928 15:47:16.592897 25732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:47:16.632808 25732 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:47:16.634934 25732 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:47:16.653609 25732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Generated new TSK 0
I20260928 15:47:16.654636 25732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:47:17.448858 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40237/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43467
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:47:17 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610437, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 15:47:17.874400 25765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:17.874903 25765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:17.875093 25765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:17.887099 25765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:17.887311 25765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:17.887506 25765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:47:17.903693 25765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.5.84:40237/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:17.905737 25765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:17.907655 25765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:17.926095 25775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:17.926433 25776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:17.935066 25765 server_base.cc:1034] running on GCE node
W20260928 15:47:17.935526 25778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:17.936450 25765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:17.937773 25765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:17.939079 25765 hybrid_clock.cc:648] HybridClock initialized: now 1790610437939012 us; error 53 us; skew 500 ppm
I20260928 15:47:17.939488 25765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:17 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610437, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:17.949713 25765 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:47:17.957805 25765 webserver.cc:460] Webserver started at http://127.13.5.65:37275/ using document root <none> and password file <none>
I20260928 15:47:17.958922 25765 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:17.959087 25765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:17.959498 25765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:47:17 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610437, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:17.979287 25765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "318b75522d16428db3f1dadffbb2fd3b"
format_stamp: "Formatted at 2026-09-28 15:47:17 on dist-test-slave-j968"
server_key: "5d8777415f05b09a3803e58d21d80899"
server_key_iv: "d4ddfd730c8ba5e9b84a867a2516b1b4"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:17.980435 25765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "318b75522d16428db3f1dadffbb2fd3b"
format_stamp: "Formatted at 2026-09-28 15:47:17 on dist-test-slave-j968"
server_key: "5d8777415f05b09a3803e58d21d80899"
server_key_iv: "d4ddfd730c8ba5e9b84a867a2516b1b4"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:17.990514 25765 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s
I20260928 15:47:18.012522 25785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:18.014922 25765 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260928 15:47:18.015177 25765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "318b75522d16428db3f1dadffbb2fd3b"
format_stamp: "Formatted at 2026-09-28 15:47:17 on dist-test-slave-j968"
server_key: "5d8777415f05b09a3803e58d21d80899"
server_key_iv: "d4ddfd730c8ba5e9b84a867a2516b1b4"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:18.015493 25765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:18.073540 25765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:18.079319 25765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:18.079748 25765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:18.083681 25765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:18.085680 25765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:18.085824 25765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:47:18.085994 25765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:18.086091 25765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:18.191987 25765 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:42513
I20260928 15:47:18.192021 25898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:42513 every 8 connection(s)
I20260928 15:47:18.193732 25765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:47:18.204505 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 25765
I20260928 15:47:18.204779 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:47:18.205792 13333 external_mini_cluster.cc:1428] Setting key 77ad5d6b752f9ab01229cfa70bf222b3
Sep 28 15:47:18 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610437, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
Sep 28 15:47:18 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610437, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:18.255530 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:18.197609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43467 (local address 127.13.5.65:40135)
0928 15:47:18.200085 (+ 2476us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:18.200156 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:18.202483 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:18.232539 (+ 30056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:18.232557 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:18.233339 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:18.234382 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:18.234418 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:18.237711 (+ 3293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:18.237720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:18.240242 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:18.240264 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:18.241454 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:18.241462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:18.241774 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:18.242613 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:18.242654 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:18.245600 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:18.250036 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:18.250061 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:18.250082 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:18.250748 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:18.251273 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:18.251278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:18.251282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:18.251573 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:18.252346 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:18.252354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:18.252836 (+ 482us) client_negotiation.cc:769] Sending connection context
0928 15:47:18.253203 (+ 367us) client_negotiation.cc:240] Negotiation successful
0928 15:47:18.253835 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2099,"thread_start_us":1860,"threads_started":1}
I20260928 15:47:18.257761 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:18.198277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:40135 (local address 127.13.5.126:43467)
0928 15:47:18.228325 (+ 30048us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:18.228333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:18.228361 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:47:18.228438 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:18.228444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:18.228536 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:18.235974 (+ 7438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:18.236230 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:18.237488 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:18.240504 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:18.241144 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:18.242270 (+ 1126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:18.245920 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:18.245927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:18.245931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:18.245988 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:18.249851 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:18.250940 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:18.250946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:18.250950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:18.251131 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:18.251731 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:18.251739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:18.251745 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:18.252179 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:18.256988 (+ 4809us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:18.257152 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 15:47:18.257396 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29886,"thread_start_us":32291,"threads_started":1}
I20260928 15:47:18.259660 25899 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43467
I20260928 15:47:18.260038 25899 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:18.261053 25899 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:47:18.264153 25684 ts_manager.cc:194] Registered new tserver with Master: 318b75522d16428db3f1dadffbb2fd3b (127.13.5.65:42513)
I20260928 15:47:18.266325 25684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:40135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:47:18.308439 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40237/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43467
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:47:18 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610437, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
W20260928 15:47:18.842913 25905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:18.843295 25905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:18.843494 25905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:18.852468 25905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:18.852639 25905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:18.852854 25905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:47:18.868517 25905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.5.84:40237/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:18.870886 25905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:18.872973 25905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:18.894523 25914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:18.895313 25916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:18.895622 25919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:18.895957 25905 server_base.cc:1034] running on GCE node
I20260928 15:47:18.896840 25905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:18.898288 25905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:18.899571 25905 hybrid_clock.cc:648] HybridClock initialized: now 1790610438899482 us; error 67 us; skew 500 ppm
I20260928 15:47:18.900048 25905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:18 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610438, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:18.904280 25905 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:47:18.906032 25905 webserver.cc:460] Webserver started at http://127.13.5.66:36539/ using document root <none> and password file <none>
I20260928 15:47:18.906776 25905 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:18.906927 25905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:18.907248 25905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:47:18 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610438, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:18.923596 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "a11f6dd4618c45d68214504ab5e27288"
format_stamp: "Formatted at 2026-09-28 15:47:18 on dist-test-slave-j968"
server_key: "9f21d3b0e28f9d772ad38a6d72802332"
server_key_iv: "f2c870ec670332102f89e4384ab5875e"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:18.924533 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "a11f6dd4618c45d68214504ab5e27288"
format_stamp: "Formatted at 2026-09-28 15:47:18 on dist-test-slave-j968"
server_key: "9f21d3b0e28f9d772ad38a6d72802332"
server_key_iv: "f2c870ec670332102f89e4384ab5875e"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:18.930540 25905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 15:47:18.956611 25929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:18.959043 25905 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.006s sys 0.005s
I20260928 15:47:18.959316 25905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "a11f6dd4618c45d68214504ab5e27288"
format_stamp: "Formatted at 2026-09-28 15:47:18 on dist-test-slave-j968"
server_key: "9f21d3b0e28f9d772ad38a6d72802332"
server_key_iv: "f2c870ec670332102f89e4384ab5875e"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:18.959661 25905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:18.998831 25905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:19.004806 25905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:19.005260 25905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:19.006589 25905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:19.008749 25905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:19.008846 25905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:19.009030 25905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:19.009099 25905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:19.060415 25905 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43275
I20260928 15:47:19.060468 26042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43275 every 8 connection(s)
I20260928 15:47:19.062142 25905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:47:19.066720 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 25905
I20260928 15:47:19.066895 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:47:19.067682 13333 external_mini_cluster.cc:1428] Setting key b50bf99ac8a5b75d00f9a04758aa0918
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Sep 28 15:47:19 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610438, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:19.109944 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:19.065384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:46299 (local address 127.13.5.126:43467)
0928 15:47:19.065880 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:19.065888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:19.081064 (+ 15176us) server_negotiation.cc:407] Connection header received
0928 15:47:19.085000 (+ 3936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:19.085007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:19.085108 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:19.085192 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:19.087530 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.088665 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.091086 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.091704 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.091802 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:19.102348 (+ 10546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:19.102355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:19.102360 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:19.102418 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:19.105862 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.107016 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.107022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.107026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.107176 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.107727 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.107732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.107736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.108139 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:19.108257 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:19.109323 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0928 15:47:19.109560 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
I20260928 15:47:19.112722 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:19.066027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43467 (local address 127.13.5.66:46299)
0928 15:47:19.080833 (+ 14806us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:19.080903 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:19.084584 (+ 3681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:19.085786 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:19.085799 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:19.086441 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:19.087297 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:19.087325 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.088892 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.088898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:19.090901 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:19.090930 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.092215 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.092223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:19.092523 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:19.096306 (+ 3783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:19.096343 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:19.102034 (+ 5691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:19.106112 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:19.106123 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:19.106144 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:19.106846 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:19.107339 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:19.107343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:19.107346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:19.107598 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:19.108386 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:19.108394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:19.108843 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 15:47:19.110392 (+ 1549us) client_negotiation.cc:240] Negotiation successful
0928 15:47:19.111055 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14423,"thread_start_us":14964,"threads_started":1}
I20260928 15:47:19.114423 26043 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43467
I20260928 15:47:19.114745 26043 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:19.115669 26043 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
I20260928 15:47:19.117477 25684 ts_manager.cc:194] Registered new tserver with Master: a11f6dd4618c45d68214504ab5e27288 (127.13.5.66:43275)
I20260928 15:47:19.118767 25684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:46299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:47:19.139424 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40237/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43467
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:47:19.270648 25899 heartbeater.cc:503] Master 127.13.5.126:43467 was elected leader, sending a full tablet report...
W20260928 15:47:19.501056 26049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:19.501410 26049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:19.501574 26049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:19.509349 26049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:19.509517 26049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:19.509689 26049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:47:19.519671 26049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.5.84:40237/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:43467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:19.521092 26049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:19.522497 26049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:19.536348 26054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:19.536537 26055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:19.536501 26057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:19.537169 26049 server_base.cc:1034] running on GCE node
I20260928 15:47:19.537866 26049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:19.538868 26049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:19.540098 26049 hybrid_clock.cc:648] HybridClock initialized: now 1790610439540071 us; error 52 us; skew 500 ppm
I20260928 15:47:19.540442 26049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:19 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610439, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:19.543676 26049 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:47:19.545205 26049 webserver.cc:460] Webserver started at http://127.13.5.67:35461/ using document root <none> and password file <none>
I20260928 15:47:19.545974 26049 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:19.546128 26049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:19.546443 26049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:47:19 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610439, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:19.561254 26049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "0f5f87a26a204961b681d19f0b1393ee"
format_stamp: "Formatted at 2026-09-28 15:47:19 on dist-test-slave-j968"
server_key: "111d82d9496b84217322789b8cf938d4"
server_key_iv: "b13cfea3c68d9b2c938214958ebf709f"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:19.562218 26049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "0f5f87a26a204961b681d19f0b1393ee"
format_stamp: "Formatted at 2026-09-28 15:47:19 on dist-test-slave-j968"
server_key: "111d82d9496b84217322789b8cf938d4"
server_key_iv: "b13cfea3c68d9b2c938214958ebf709f"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:19.568177 26049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:47:19.584379 26064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:19.586199 26049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:47:19.586418 26049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "0f5f87a26a204961b681d19f0b1393ee"
format_stamp: "Formatted at 2026-09-28 15:47:19 on dist-test-slave-j968"
server_key: "111d82d9496b84217322789b8cf938d4"
server_key_iv: "b13cfea3c68d9b2c938214958ebf709f"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:19.586681 26049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:19.616340 26049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:19.620186 26049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:19.620520 26049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:19.621503 26049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:19.622985 26049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:19.623097 26049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:19.623255 26049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:19.623334 26049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:19.659173 26049 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:44119
I20260928 15:47:19.659211 26177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:44119 every 8 connection(s)
I20260928 15:47:19.660591 26049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:47:19.662966 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 26049
I20260928 15:47:19.663193 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:47:19.663658 13333 external_mini_cluster.cc:1428] Setting key 3b37a8f36341ae0b590852b1a6d312fe
Sep 28 15:47:19 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610439, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:19.682372 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:19.664213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:50745 (local address 127.13.5.126:43467)
0928 15:47:19.668353 (+ 4140us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:19.668359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:19.668381 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:47:19.668479 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:19.668482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:19.668548 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:19.668633 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:19.670036 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.670736 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.672368 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.672778 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.672869 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:19.676129 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:19.676134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:19.676137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:19.676172 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:19.679428 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.680302 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.680306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.680308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.680451 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.680850 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.680854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.680856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.681156 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:19.681276 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:19.681922 (+ 646us) server_negotiation.cc:299] Negotiation successful
0928 15:47:19.682090 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3999,"thread_start_us":3794,"threads_started":1}
I20260928 15:47:19.683357 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:19.663960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43467 (local address 127.13.5.67:50745)
0928 15:47:19.665052 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:19.665121 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:19.668193 (+ 3072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:19.668868 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:19.668876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:19.669294 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:19.669891 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:19.669909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.670870 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.670874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:19.672229 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:19.672248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.672958 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.672963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:19.673165 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:19.673795 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:19.673818 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:19.675855 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:19.679602 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:19.679616 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:19.679636 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:19.680176 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:19.680548 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:19.680552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:19.680554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:19.680750 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:19.681261 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:19.681267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:19.681600 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 15:47:19.681832 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 15:47:19.682228 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":407,"threads_started":1}
I20260928 15:47:19.685070 26178 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43467
I20260928 15:47:19.685397 26178 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:19.686115 26178 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
I20260928 15:47:19.687584 25684 ts_manager.cc:194] Registered new tserver with Master: 0f5f87a26a204961b681d19f0b1393ee (127.13.5.67:44119)
I20260928 15:47:19.688428 25684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:50745
I20260928 15:47:19.692325 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:47:19.693925 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 25765
Sep 28 15:47:19 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:47:19.710407 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:19.694785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.13.5.65:42513)
0928 15:47:19.695394 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:19.695401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:19.695426 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:47:19.695485 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:19.695491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:19.695693 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:19.695834 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:19.697445 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.698487 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.700027 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.700570 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.700674 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:19.704906 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:19.704917 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:19.704937 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:19.704978 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:19.707737 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.708491 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.708498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.708501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.708650 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.709015 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.709018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.709020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.709356 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:19.709656 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:19.709897 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 15:47:19.710117 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":293,"threads_started":1}
I20260928 15:47:19.711727 25894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:19.797506 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 25765
I20260928 15:47:19.814213 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 25905
Sep 28 15:47:19 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:47:19.832136 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:19.815381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.13.5.66:43275)
0928 15:47:19.816042 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:19.816050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:19.816079 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:47:19.816157 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:19.816163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:19.816456 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:19.816610 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:19.817908 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.819113 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.820726 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.821317 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.821535 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:19.826138 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:19.826151 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:19.826168 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:19.826208 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:19.829086 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.829835 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.829843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.829846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.829991 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.830405 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.830412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.830415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.830788 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:19.831115 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:19.831443 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 15:47:19.831744 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":287,"threads_started":1}
I20260928 15:47:19.833503 26038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:19.917266 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 25905
I20260928 15:47:19.935910 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 26049
Sep 28 15:47:19 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:47:19.953176 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:19.937069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.13.5.67:44119)
0928 15:47:19.937728 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:19.937735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:19.937762 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:47:19.937822 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:19.937829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:19.938059 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:19.938236 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:19.939621 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.940753 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.942094 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:19.942686 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:19.942829 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:19.947314 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:19.947325 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:19.947341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:19.947382 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:19.950087 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.950763 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.950770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.950773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.950905 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:19.951308 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:19.951313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:19.951316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:19.951670 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:19.952068 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:19.952520 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 15:47:19.952801 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":314,"threads_started":1}
I20260928 15:47:19.954543 26173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:20.040134 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 26049
I20260928 15:47:20.056838 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 25652
I20260928 15:47:20.058326 25715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:20.169365 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 25652
I20260928 15:47:20.192693 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43467
--webserver_interface=127.13.5.126
--webserver_port=37687
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43467
--ranger_config_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40237/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.5.84:40237/kms with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:20.526806 26190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:20.527171 26190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:20.527331 26190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:20.534749 26190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:20.534895 26190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:20.535007 26190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:20.535110 26190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:20.535203 26190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:20.545672 26190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.5.84:40237/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43467
--ranger_config_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:20.547086 26190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:20.548501 26190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:20.562353 26195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:20.562273 26196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:20.562300 26198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:20.563241 26190 server_base.cc:1034] running on GCE node
I20260928 15:47:20.564129 26190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:20.565624 26190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:20.566835 26190 hybrid_clock.cc:648] HybridClock initialized: now 1790610440566795 us; error 32 us; skew 500 ppm
I20260928 15:47:20.567173 26190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:20 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610440, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:20.570324 26190 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:47:20.571805 26190 webserver.cc:460] Webserver started at http://127.13.5.126:37687/ using document root <none> and password file <none>
I20260928 15:47:20.572531 26190 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:20.572649 26190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:47:20 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610440, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:20.626349 26190 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:47:20.629761 26206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:20.631364 26190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:47:20.631573 26190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "d094f2bceb4a4894aaedad432f4d3eb1"
format_stamp: "Formatted at 2026-09-28 15:47:16 on dist-test-slave-j968"
server_key: "c97ba6c72ac38ff6ee02fe78f5fd0e7d"
server_key_iv: "a1bb700d75f9b7fbaf3652d6adf58d06"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:20.632689 26190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:20.650939 26190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:20.655937 26190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:20.656296 26190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:20.675726 26190 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43467
I20260928 15:47:20.675705 26257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43467 every 8 connection(s)
I20260928 15:47:20.677469 26190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:47:20.684751 26258 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:47:20.687983 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 26190
I20260928 15:47:20.688324 26258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Bootstrap starting.
I20260928 15:47:20.688905 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40237/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:42513
--local_ip_for_outbound_sockets=127.13.5.65
--tserver_master_addrs=127.13.5.126:43467
--webserver_port=37275
--webserver_interface=127.13.5.65
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.5.84:40237/kms with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:47:20.708268 26258 log.cc:824] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:20.717665 26258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:47:20.718216 26258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Bootstrap complete.
I20260928 15:47:20.727766 26258 raft_consensus.cc:348] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } }
I20260928 15:47:20.728740 26258 raft_consensus.cc:729] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d094f2bceb4a4894aaedad432f4d3eb1, State: Initialized, Role: FOLLOWER
I20260928 15:47:20.729403 26258 consensus_queue.cc:260] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } }
I20260928 15:47:20.729651 26258 raft_consensus.cc:388] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:47:20.729771 26258 raft_consensus.cc:482] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:47:20.729954 26258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:47:20.733928 26258 raft_consensus.cc:504] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } }
I20260928 15:47:20.734428 26258 leader_election.cc:302] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d094f2bceb4a4894aaedad432f4d3eb1; no voters:
I20260928 15:47:20.735071 26258 leader_election.cc:288] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:47:20.735270 26262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:47:20.736400 26262 raft_consensus.cc:686] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [term 2 LEADER]: Becoming Leader. State: Replica: d094f2bceb4a4894aaedad432f4d3eb1, State: Running, Role: LEADER
I20260928 15:47:20.737089 26262 consensus_queue.cc:237] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } }
I20260928 15:47:20.737329 26258 sys_catalog.cc:564] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:20.740135 26264 sys_catalog.cc:455] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d094f2bceb4a4894aaedad432f4d3eb1. Latest consensus state: current_term: 2 leader_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } } }
I20260928 15:47:20.740227 26263 sys_catalog.cc:455] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d094f2bceb4a4894aaedad432f4d3eb1" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43467 } } }
I20260928 15:47:20.740594 26264 sys_catalog.cc:458] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:20.740660 26263 sys_catalog.cc:458] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:20.741818 26268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:47:20.759681 26268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:47:20.761380 26268 catalog_manager.cc:1259] Loaded cluster ID: c0ddf69cfc444b0c825941586777a9a5
I20260928 15:47:20.761576 26268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:47:20.762562 26258 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:47:20.771283 26268 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:47:20.774214 26268 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Loaded TSK: 0
I20260928 15:47:20.776260 26268 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:47:21.167964 26260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:21.168540 26260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:21.168782 26260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:21.181119 26260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:21.181387 26260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:21.181605 26260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:47:21.198300 26260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.5.84:40237/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:42513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=37275
--tserver_master_addrs=127.13.5.126:43467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:21.200285 26260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:21.202241 26260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:21.225206 26295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:21.226306 26296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:21.237876 26299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:21.238579 26260 server_base.cc:1034] running on GCE node
I20260928 15:47:21.239547 26260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:21.243232 26260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:21.244577 26260 hybrid_clock.cc:648] HybridClock initialized: now 1790610441244429 us; error 144 us; skew 500 ppm
I20260928 15:47:21.245153 26260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:21 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610441, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:21.249467 26260 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:47:21.251652 26260 webserver.cc:460] Webserver started at http://127.13.5.65:37275/ using document root <none> and password file <none>
I20260928 15:47:21.252707 26260 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:21.252874 26260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:47:21 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610441, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:21.293699 26260 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:47:21.297660 26306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:21.299558 26260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:47:21.299826 26260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "318b75522d16428db3f1dadffbb2fd3b"
format_stamp: "Formatted at 2026-09-28 15:47:17 on dist-test-slave-j968"
server_key: "5d8777415f05b09a3803e58d21d80899"
server_key_iv: "d4ddfd730c8ba5e9b84a867a2516b1b4"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:21.301312 26260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:21.339377 26260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:21.345124 26260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:21.345522 26260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:21.346873 26260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:21.348851 26260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:21.348974 26260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:21.349107 26260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:21.349187 26260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:21.404346 26260 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:42513
I20260928 15:47:21.404454 26421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:42513 every 8 connection(s)
I20260928 15:47:21.406219 26260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:47:21.409562 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 26260
I20260928 15:47:21.410405 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40237/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:43275
--local_ip_for_outbound_sockets=127.13.5.66
--tserver_master_addrs=127.13.5.126:43467
--webserver_port=36539
--webserver_interface=127.13.5.66
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.5.84:40237/kms with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:47:21 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610441, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:21.465523 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:21.426505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:58697 (local address 127.13.5.126:43467)
0928 15:47:21.430352 (+ 3847us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:21.430377 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:21.441186 (+ 10809us) server_negotiation.cc:407] Connection header received
0928 15:47:21.445441 (+ 4255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:21.445464 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:21.446021 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:21.446433 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:21.448985 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:21.450615 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:21.452554 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:21.453396 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:21.453525 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:21.457404 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:21.457412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:21.457428 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:21.457469 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:21.460698 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:21.461497 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:21.461504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:21.461511 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:21.461658 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:21.462131 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:21.462136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:21.462138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:21.462659 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:21.462882 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:21.463671 (+ 789us) server_negotiation.cc:299] Negotiation successful
0928 15:47:21.464166 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3530,"thread_start_us":3297,"threads_started":1}
I20260928 15:47:21.465548 26426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:21.426706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43467 (local address 127.13.5.65:58697)
0928 15:47:21.440889 (+ 14183us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:21.440968 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:21.444613 (+ 3645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:21.447307 (+ 2694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:21.447319 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:21.448014 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:21.448737 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:21.448756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:21.450815 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:21.450823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:21.452383 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:21.452399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:21.453910 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:21.453915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:21.454112 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:21.454845 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:21.454873 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:21.457012 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:21.460851 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:21.460861 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:21.460877 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:21.461313 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:21.461780 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:21.461785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:21.461789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:21.462003 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:21.462799 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:21.462807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:21.463156 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 15:47:21.463431 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 15:47:21.464098 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13679,"thread_start_us":16294,"threads_started":1}
I20260928 15:47:21.469024 26422 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43467
I20260928 15:47:21.469357 26422 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:21.470110 26422 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
W20260928 15:47:21.473366 26422 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:43467 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:47:21.473610 26422 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:21.473891 26422 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
I20260928 15:47:21.475222 26422 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:21.475509 26422 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
W20260928 15:47:21.476676 26422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:47:21.867096 26425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:21.867540 26425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:21.867764 26425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:21.879664 26425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:21.879879 26425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:21.880091 26425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:47:21.892563 26425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.5.84:40237/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:43275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=36539
--tserver_master_addrs=127.13.5.126:43467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:21.894047 26425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:21.895732 26425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:21.937630 26440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:21.946200 26441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:21.946563 26425 server_base.cc:1034] running on GCE node
W20260928 15:47:21.947124 26445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:21.948125 26425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:21.956809 26425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:21.958164 26425 hybrid_clock.cc:648] HybridClock initialized: now 1790610441958091 us; error 62 us; skew 500 ppm
I20260928 15:47:21.959215 26425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:21 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610441, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:21.964004 26425 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:47:21.973042 26425 webserver.cc:460] Webserver started at http://127.13.5.66:36539/ using document root <none> and password file <none>
I20260928 15:47:21.973927 26425 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:21.974118 26425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:47:21 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610441, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:22.013907 26425 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.004s sys 0.010s
I20260928 15:47:22.017536 26453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:22.019404 26425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:47:22.019719 26425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "a11f6dd4618c45d68214504ab5e27288"
format_stamp: "Formatted at 2026-09-28 15:47:18 on dist-test-slave-j968"
server_key: "9f21d3b0e28f9d772ad38a6d72802332"
server_key_iv: "f2c870ec670332102f89e4384ab5875e"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:22.021230 26425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 15:47:22 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610442, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:22.045301 26425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:22.051394 26425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:22.051846 26425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:22.055981 26425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:22.072891 26425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:22.073117 26425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:47:22.073273 26425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:22.073382 26425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:22.128845 26425 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43275
I20260928 15:47:22.128926 26568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43275 every 8 connection(s)
I20260928 15:47:22.130849 26425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:47:22.140502 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 26425
I20260928 15:47:22.141171 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.5.84:40237/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:44119
--local_ip_for_outbound_sockets=127.13.5.67
--tserver_master_addrs=127.13.5.126:43467
--webserver_port=35461
--webserver_interface=127.13.5.67
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.5.84:40237/kms with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:47:22 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610441, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:22.176734 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:22.134279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:39847 (local address 127.13.5.126:43467)
0928 15:47:22.140306 (+ 6027us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:22.140316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:22.152038 (+ 11722us) server_negotiation.cc:407] Connection header received
0928 15:47:22.155229 (+ 3191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:22.155236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:22.155343 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:22.155473 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:22.157918 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:22.159588 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:22.162253 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:22.162945 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:22.163078 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:22.167824 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:22.167831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:22.167836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:22.167917 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:22.171765 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:22.172849 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:22.172854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:22.172858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:22.173009 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:22.173627 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:22.173633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:22.173636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:22.174077 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:22.174242 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:22.176067 (+ 1825us) server_negotiation.cc:299] Negotiation successful
0928 15:47:22.176322 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5823,"thread_start_us":5715,"threads_started":1}
I20260928 15:47:22.178834 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:22.136134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43467 (local address 127.13.5.66:39847)
0928 15:47:22.151765 (+ 15631us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:22.151838 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:22.154780 (+ 2942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:22.156023 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:22.156036 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:22.156736 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:22.157650 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:22.157679 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:22.159799 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:22.159805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:22.162074 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:22.162088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:22.163458 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:22.163466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:22.163682 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:22.164568 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:22.164617 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:22.167520 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:22.171972 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:22.171983 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:22.172005 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:22.172660 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:22.173197 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:22.173202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:22.173205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:22.173459 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:22.174322 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:22.174331 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:22.174838 (+ 507us) client_negotiation.cc:769] Sending connection context
0928 15:47:22.175191 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 15:47:22.175850 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15250,"thread_start_us":14942,"threads_started":1}
I20260928 15:47:22.180964 26569 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43467
I20260928 15:47:22.181340 26569 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:22.182385 26569 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
I20260928 15:47:22.184876 26223 ts_manager.cc:194] Registered new tserver with Master: a11f6dd4618c45d68214504ab5e27288 (127.13.5.66:43275)
I20260928 15:47:22.187109 26223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:39847
I20260928 15:47:22.476845 26422 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:22.477319 26422 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
I20260928 15:47:22.478921 26223 ts_manager.cc:194] Registered new tserver with Master: 318b75522d16428db3f1dadffbb2fd3b (127.13.5.65:42513)
I20260928 15:47:22.480189 26223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:58697
Sep 28 15:47:22 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610442, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
W20260928 15:47:22.702226 26574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:22.702694 26574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:22.702886 26574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:22.714053 26574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:22.714255 26574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:22.714462 26574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:47:22.727814 26574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.5.84:40237/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:44119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=35461
--tserver_master_addrs=127.13.5.126:43467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:22.729339 26574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:22.730897 26574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:22.748224 26579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:22.748302 26580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:22.752238 26582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:22.754159 26574 server_base.cc:1034] running on GCE node
I20260928 15:47:22.754830 26574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:22.755908 26574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:22.757164 26574 hybrid_clock.cc:648] HybridClock initialized: now 1790610442757089 us; error 65 us; skew 500 ppm
I20260928 15:47:22.757508 26574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:22 dist-test-slave-j968 krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610442, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:22.761788 26574 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:47:22.765460 26574 webserver.cc:460] Webserver started at http://127.13.5.67:35461/ using document root <none> and password file <none>
I20260928 15:47:22.766384 26574 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:22.766530 26574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:47:22 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610442, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:22.813486 26574 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.002s
I20260928 15:47:22.817668 26590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:22.819821 26574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260928 15:47:22.820052 26574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "0f5f87a26a204961b681d19f0b1393ee"
format_stamp: "Formatted at 2026-09-28 15:47:19 on dist-test-slave-j968"
server_key: "111d82d9496b84217322789b8cf938d4"
server_key_iv: "b13cfea3c68d9b2c938214958ebf709f"
server_key_version: "kuduclusterkey@0"
I20260928 15:47:22.821631 26574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:22.894073 26574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:22.900081 26574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:22.900514 26574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:22.902230 26574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:22.909183 26574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:22.909384 26574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:47:22.909559 26574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:22.909674 26574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:47:22 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610442, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for HTTP/127.13.5.84@KRBTEST.COM
I20260928 15:47:23.079810 26703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:44119 every 8 connection(s)
I20260928 15:47:23.080214 26574 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:44119
I20260928 15:47:23.082406 26574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:47:23.083444 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 26574
I20260928 15:47:23.144726 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.085103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.13.5.126:43467)
0928 15:47:23.099017 (+ 13914us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.099029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.104605 (+ 5576us) server_negotiation.cc:407] Connection header received
0928 15:47:23.104697 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.104706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.104841 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.104974 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:23.108053 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.109724 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.116728 (+ 7004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.117688 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.117869 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.135264 (+ 17395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:23.135273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:23.135279 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:23.135335 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:23.140256 (+ 4921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.141351 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.141358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.141380 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.141563 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.142976 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.142982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.142986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.143342 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:23.143671 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.144040 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.144302 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13694,"thread_start_us":13927,"threads_started":1}
Sep 28 15:47:23 dist-test-slave-j968 krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610442, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:23.157413 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.104576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:37793 (local address 127.13.5.126:43467)
0928 15:47:23.119944 (+ 15368us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.119960 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.123924 (+ 3964us) server_negotiation.cc:407] Connection header received
0928 15:47:23.125527 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.125534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.125648 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.125738 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:23.137511 (+ 11773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.139225 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.141850 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.142612 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.142754 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.148606 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:23.148614 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:23.148620 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:23.148671 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:23.152642 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.153777 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.153784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.153788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.153944 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.154593 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.154598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.154602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.154990 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:23.155157 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.156736 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.156993 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15178,"thread_start_us":15384,"threads_started":1}
I20260928 15:47:23.160710 26709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.103829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43467 (local address 127.13.5.67:37793)
0928 15:47:23.121201 (+ 17372us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.121276 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:23.125156 (+ 3880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:23.126448 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:23.126463 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:23.136099 (+ 9636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:23.137179 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.137213 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.139720 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.139726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.141683 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.141702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.142894 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.142902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.143209 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.145327 (+ 2118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:23.145370 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:23.148266 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:23.152879 (+ 4613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:23.152891 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:23.152911 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:23.153596 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:23.154109 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:23.154114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:23.154117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:23.154440 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:23.155236 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:23.155246 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:23.155739 (+ 493us) client_negotiation.cc:769] Sending connection context
0928 15:47:23.156139 (+ 400us) client_negotiation.cc:240] Negotiation successful
0928 15:47:23.158156 (+ 2017us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16936,"thread_start_us":19951,"threads_started":1}
I20260928 15:47:23.162966 26704 heartbeater.cc:347] Connected to a master server at 127.13.5.126:43467
I20260928 15:47:23.163378 26704 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:23.164472 26704 heartbeater.cc:511] Master 127.13.5.126:43467 requested a full tablet report, sending...
I20260928 15:47:23.166319 26223 ts_manager.cc:194] Registered new tserver with Master: 0f5f87a26a204961b681d19f0b1393ee (127.13.5.67:44119)
I20260928 15:47:23.167472 26223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:37793
I20260928 15:47:23.173544 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:47:23.192548 26569 heartbeater.cc:503] Master 127.13.5.126:43467 was elected leader, sending a full tablet report...
I20260928 15:47:23.204445 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.177885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.13.5.126:43467)
0928 15:47:23.178059 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.178066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.188737 (+ 10671us) server_negotiation.cc:407] Connection header received
0928 15:47:23.189310 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.189317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.189411 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.189520 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:23.192029 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.193023 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.195428 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.195938 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.196087 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.198651 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:23.198657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:23.198663 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:23.198706 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:23.201552 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.202259 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.202265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.202270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.202398 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.202777 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.202783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.202786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.203075 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:23.203323 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.203685 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.203941 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:47:23.209914 26223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:47:23.217681 26223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:47:23.258872 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.234931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:44119 (local address 127.0.0.1:35304)
0928 15:47:23.248499 (+ 13568us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.248551 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:23.248749 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:23.250461 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:23.250469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:23.250503 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:23.250944 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.250960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.253057 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.253062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.255534 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.255551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.257821 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.257827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.258050 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.258109 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 15:47:23.258254 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 15:47:23.258468 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13229,"thread_start_us":13094,"threads_started":1}
I20260928 15:47:23.260066 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.235614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.13.5.67:44119)
0928 15:47:23.249448 (+ 13834us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.249458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.249494 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:47:23.249620 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.249629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.249907 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.250138 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:23.251172 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.252855 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.255728 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.257280 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.257408 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.257658 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.259456 (+ 1798us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.259633 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13639,"thread_start_us":13830,"threads_started":1}
I20260928 15:47:23.266131 26639 tablet_service.cc:1467] Processing CreateTablet for tablet b1ab94077a1e494fb8a9c5047a6c309f (DEFAULT_TABLE table=test-table [id=a11fc686f237414aac4fb0f985d44ebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:23.267676 26639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ab94077a1e494fb8a9c5047a6c309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:23.271713 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.241163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57234)
0928 15:47:23.259146 (+ 17983us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.259167 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:23.259278 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:23.263912 (+ 4634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:23.263918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:23.263957 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:23.264142 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.264151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.266087 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.266093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.268511 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.268543 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.270778 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.270784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.270971 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.271039 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 15:47:23.271133 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 15:47:23.271320 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17881,"thread_start_us":25985,"threads_started":1}
I20260928 15:47:23.273736 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.241389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.13.5.65:42513)
0928 15:47:23.263124 (+ 21735us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.263136 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.263172 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:47:23.263298 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.263310 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.263634 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.263883 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:23.264359 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.265894 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.268772 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.270556 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.272420 (+ 1864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.272809 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.272940 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.273127 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21506,"thread_start_us":22729,"threads_started":1}
I20260928 15:47:23.278225 26355 tablet_service.cc:1467] Processing CreateTablet for tablet b1ab94077a1e494fb8a9c5047a6c309f (DEFAULT_TABLE table=test-table [id=a11fc686f237414aac4fb0f985d44ebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:23.280144 26355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ab94077a1e494fb8a9c5047a6c309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:23.290242 26729 tablet_bootstrap.cc:492] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Bootstrap starting.
I20260928 15:47:23.295621 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.249290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43275 (local address 127.0.0.1:41482)
0928 15:47:23.261448 (+ 12158us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.261471 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:23.261609 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:23.284700 (+ 23091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:23.284709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:23.284749 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:23.285165 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.285180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.289136 (+ 3956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.289142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.291224 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.291248 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.294708 (+ 3460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.294716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.294924 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.294974 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 15:47:23.295059 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:47:23.295230 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11931,"thread_start_us":12405,"threads_started":1}
I20260928 15:47:23.296231 26729 tablet_bootstrap.cc:654] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:23.296986 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.249605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.13.5.66:43275)
0928 15:47:23.283210 (+ 33605us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.283219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.283257 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:47:23.283389 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.283398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.283856 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.286951 (+ 3095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:23.287267 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.288934 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.291573 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.293100 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.295960 (+ 2860us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.296240 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.296329 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.296525 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33408,"thread_start_us":34445,"threads_started":1}
I20260928 15:47:23.298625 26729 log.cc:824] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:23.301781 26503 tablet_service.cc:1467] Processing CreateTablet for tablet b1ab94077a1e494fb8a9c5047a6c309f (DEFAULT_TABLE table=test-table [id=a11fc686f237414aac4fb0f985d44ebd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:23.303942 26503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1ab94077a1e494fb8a9c5047a6c309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:23.306457 26729 tablet_bootstrap.cc:492] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: No bootstrap required, opened a new log
I20260928 15:47:23.306782 26729 ts_tablet_manager.cc:1397] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260928 15:47:23.309054 26735 tablet_bootstrap.cc:492] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: Bootstrap starting.
I20260928 15:47:23.313056 26735 tablet_bootstrap.cc:654] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:23.314497 26735 log.cc:824] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:23.317806 26729 raft_consensus.cc:348] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.318521 26729 raft_consensus.cc:374] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:23.318691 26729 raft_consensus.cc:729] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f5f87a26a204961b681d19f0b1393ee, State: Initialized, Role: FOLLOWER
I20260928 15:47:23.319541 26729 consensus_queue.cc:260] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.326025 26738 tablet_bootstrap.cc:492] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Bootstrap starting.
I20260928 15:47:23.326638 26704 heartbeater.cc:503] Master 127.13.5.126:43467 was elected leader, sending a full tablet report...
I20260928 15:47:23.326890 26729 ts_tablet_manager.cc:1428] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Time spent starting tablet: real 0.020s user 0.016s sys 0.005s
I20260928 15:47:23.328025 26735 tablet_bootstrap.cc:492] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: No bootstrap required, opened a new log
I20260928 15:47:23.328399 26735 ts_tablet_manager.cc:1397] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20260928 15:47:23.335548 26738 tablet_bootstrap.cc:654] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:23.337081 26738 log.cc:824] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:23.338680 26735 raft_consensus.cc:348] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.339175 26735 raft_consensus.cc:374] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:23.339299 26735 raft_consensus.cc:729] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318b75522d16428db3f1dadffbb2fd3b, State: Initialized, Role: FOLLOWER
I20260928 15:47:23.340063 26735 consensus_queue.cc:260] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.340687 26738 tablet_bootstrap.cc:492] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: No bootstrap required, opened a new log
I20260928 15:47:23.341030 26738 ts_tablet_manager.cc:1397] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260928 15:47:23.341185 26422 heartbeater.cc:503] Master 127.13.5.126:43467 was elected leader, sending a full tablet report...
I20260928 15:47:23.341871 26735 ts_tablet_manager.cc:1428] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 15:47:23.351897 26738 raft_consensus.cc:348] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.352488 26738 raft_consensus.cc:374] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:23.352602 26738 raft_consensus.cc:729] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11f6dd4618c45d68214504ab5e27288, State: Initialized, Role: FOLLOWER
I20260928 15:47:23.353348 26738 consensus_queue.cc:260] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.355283 26738 ts_tablet_manager.cc:1428] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260928 15:47:23.367777 26708 tablet.cc:2369] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:47:23.390566 26571 tablet.cc:2369] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:47:23.445658 26423 tablet.cc:2369] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:47:23.467549 26740 raft_consensus.cc:482] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:47:23.467975 26740 raft_consensus.cc:504] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.469863 26740 leader_election.cc:288] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f5f87a26a204961b681d19f0b1393ee (127.13.5.67:44119), a11f6dd4618c45d68214504ab5e27288 (127.13.5.66:43275)
I20260928 15:47:23.477955 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.470248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:46743 (local address 127.13.5.67:44119)
0928 15:47:23.470454 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.470460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.470886 (+ 426us) server_negotiation.cc:407] Connection header received
0928 15:47:23.471239 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.471248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.471357 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.471450 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:23.472389 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.473523 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.476170 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.477019 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.477174 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.477282 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.477561 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.477687 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:47:23.478010 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.470119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:44119 (local address 127.13.5.65:46743)
0928 15:47:23.470775 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.470797 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:23.471013 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:23.471713 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:23.471718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:23.471750 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:23.472226 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.472240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.473701 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.473706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.475998 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.476013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.477228 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.477234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.477395 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.477425 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:47:23.477525 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:47:23.477656 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":328,"threads_started":1}
I20260928 15:47:23.478741 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.471210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:53625 (local address 127.13.5.66:43275)
0928 15:47:23.471393 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.471399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.472106 (+ 707us) server_negotiation.cc:407] Connection header received
0928 15:47:23.472290 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.472298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.472414 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.472496 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:23.473356 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.474327 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.476384 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.477615 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.477743 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.477842 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.478382 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.478513 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:47:23.479142 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ab94077a1e494fb8a9c5047a6c309f" candidate_uuid: "318b75522d16428db3f1dadffbb2fd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f87a26a204961b681d19f0b1393ee" is_pre_election: true
I20260928 15:47:23.479471 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.470941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43275 (local address 127.13.5.65:53625)
0928 15:47:23.471969 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.472006 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:23.472144 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:23.472655 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:23.472660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:23.472683 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:23.473047 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.473177 (+ 130us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.474468 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.474476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.476217 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:23.476230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.478017 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.478023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:23.478200 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.478233 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:47:23.478974 (+ 741us) client_negotiation.cc:240] Negotiation successful
0928 15:47:23.479119 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":4,"thread_start_us":646,"threads_started":1}
I20260928 15:47:23.479698 26659 raft_consensus.cc:2454] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 318b75522d16428db3f1dadffbb2fd3b in term 0.
I20260928 15:47:23.480451 26309 leader_election.cc:302] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5f87a26a204961b681d19f0b1393ee, 318b75522d16428db3f1dadffbb2fd3b; no voters:
I20260928 15:47:23.480526 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ab94077a1e494fb8a9c5047a6c309f" candidate_uuid: "318b75522d16428db3f1dadffbb2fd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11f6dd4618c45d68214504ab5e27288" is_pre_election: true
I20260928 15:47:23.480926 26740 raft_consensus.cc:2781] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:47:23.481076 26740 raft_consensus.cc:482] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:47:23.481058 26523 raft_consensus.cc:2454] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 318b75522d16428db3f1dadffbb2fd3b in term 0.
I20260928 15:47:23.481168 26740 raft_consensus.cc:3037] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:23.484313 26740 raft_consensus.cc:504] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.484997 26740 leader_election.cc:288] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [CANDIDATE]: Term 1 election: Requested vote from peers 0f5f87a26a204961b681d19f0b1393ee (127.13.5.67:44119), a11f6dd4618c45d68214504ab5e27288 (127.13.5.66:43275)
I20260928 15:47:23.485479 26659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ab94077a1e494fb8a9c5047a6c309f" candidate_uuid: "318b75522d16428db3f1dadffbb2fd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f5f87a26a204961b681d19f0b1393ee"
I20260928 15:47:23.485616 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1ab94077a1e494fb8a9c5047a6c309f" candidate_uuid: "318b75522d16428db3f1dadffbb2fd3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11f6dd4618c45d68214504ab5e27288"
I20260928 15:47:23.485709 26659 raft_consensus.cc:3037] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:23.485821 26523 raft_consensus.cc:3037] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:23.489048 26659 raft_consensus.cc:2454] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 318b75522d16428db3f1dadffbb2fd3b in term 1.
I20260928 15:47:23.489040 26523 raft_consensus.cc:2454] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 318b75522d16428db3f1dadffbb2fd3b in term 1.
I20260928 15:47:23.489487 26309 leader_election.cc:302] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f5f87a26a204961b681d19f0b1393ee, 318b75522d16428db3f1dadffbb2fd3b; no voters:
I20260928 15:47:23.489882 26740 raft_consensus.cc:2781] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:47:23.490460 26740 raft_consensus.cc:686] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 1 LEADER]: Becoming Leader. State: Replica: 318b75522d16428db3f1dadffbb2fd3b, State: Running, Role: LEADER
I20260928 15:47:23.491067 26740 consensus_queue.cc:237] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } }
I20260928 15:47:23.496070 26223 catalog_manager.cc:5495] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 318b75522d16428db3f1dadffbb2fd3b (127.13.5.65). New cstate: current_term: 1 leader_uuid: "318b75522d16428db3f1dadffbb2fd3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "318b75522d16428db3f1dadffbb2fd3b" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 42513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 } health_report { overall_health: UNKNOWN } } }
I20260928 15:47:23.544303 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.538056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.13.5.65:42513)
0928 15:47:23.538272 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.538278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.538298 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:47:23.538411 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.538418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.538511 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.538669 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:47:23.539055 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.539928 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.541969 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.542425 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.542527 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.543108 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:47:23.543676 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:47:23.543786 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.543958 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.544027 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:47:23.554414 26523 raft_consensus.cc:1261] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 1 FOLLOWER]: Refusing update from remote peer 318b75522d16428db3f1dadffbb2fd3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:47:23.554414 26659 raft_consensus.cc:1261] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 1 FOLLOWER]: Refusing update from remote peer 318b75522d16428db3f1dadffbb2fd3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:47:23.555491 26744 consensus_queue.cc:1035] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11f6dd4618c45d68214504ab5e27288" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:47:23.555923 26740 consensus_queue.cc:1035] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f5f87a26a204961b681d19f0b1393ee" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:47:23.585613 26749 mvcc.cc:205] Tried to move back new op lower bound from 7334340376787255296 to 7334340376543571968. Current Snapshot: MvccSnapshot[applied={T|T < 7334340376787255296}]
I20260928 15:47:23.601656 26223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:47:23.601980 26223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:47:23.605923 26223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d094f2bceb4a4894aaedad432f4d3eb1: Sending DeleteTablet for 3 replicas of tablet b1ab94077a1e494fb8a9c5047a6c309f
I20260928 15:47:23.607120 26355 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ab94077a1e494fb8a9c5047a6c309f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:23 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:57234
I20260928 15:47:23.607295 26503 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ab94077a1e494fb8a9c5047a6c309f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:23 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:41482
I20260928 15:47:23.614079 26639 tablet_service.cc:1514] Processing DeleteTablet for tablet b1ab94077a1e494fb8a9c5047a6c309f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:23 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:35304
I20260928 15:47:23.617990 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 26260
I20260928 15:47:23.623708 26757 tablet_replica.cc:330] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: stopping tablet replica
I20260928 15:47:23.624390 26757 raft_consensus.cc:2229] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:47:23.625084 26757 raft_consensus.cc:2258] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:23.627300 26758 tablet_replica.cc:330] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: stopping tablet replica
I20260928 15:47:23.627893 26758 raft_consensus.cc:2229] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:47:23.628475 26758 raft_consensus.cc:2258] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:23.629156 26756 tablet_replica.cc:330] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: stopping tablet replica
I20260928 15:47:23.629431 26757 ts_tablet_manager.cc:1905] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:23.629642 26756 raft_consensus.cc:2229] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:47:23.630458 26756 raft_consensus.cc:2258] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:23.632737 26758 ts_tablet_manager.cc:1905] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:23.636822 26757 ts_tablet_manager.cc:1918] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:47:23.637061 26757 log.cc:1196] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/b1ab94077a1e494fb8a9c5047a6c309f
I20260928 15:47:23.637742 26757 ts_tablet_manager.cc:1939] T b1ab94077a1e494fb8a9c5047a6c309f P a11f6dd4618c45d68214504ab5e27288: Deleting consensus metadata
I20260928 15:47:23.639719 26209 catalog_manager.cc:4841] TS a11f6dd4618c45d68214504ab5e27288 (127.13.5.66:43275): tablet b1ab94077a1e494fb8a9c5047a6c309f (table test-table [id=a11fc686f237414aac4fb0f985d44ebd]) successfully deleted
I20260928 15:47:23.640229 26758 ts_tablet_manager.cc:1918] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:47:23.640619 26758 log.cc:1196] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/b1ab94077a1e494fb8a9c5047a6c309f
I20260928 15:47:23.641244 26758 ts_tablet_manager.cc:1939] T b1ab94077a1e494fb8a9c5047a6c309f P 0f5f87a26a204961b681d19f0b1393ee: Deleting consensus metadata
I20260928 15:47:23.641773 26756 ts_tablet_manager.cc:1905] T b1ab94077a1e494fb8a9c5047a6c309f P 318b75522d16428db3f1dadffbb2fd3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:23.643159 26209 catalog_manager.cc:4841] TS 0f5f87a26a204961b681d19f0b1393ee (127.13.5.67:44119): tablet b1ab94077a1e494fb8a9c5047a6c309f (table test-table [id=a11fc686f237414aac4fb0f985d44ebd]) successfully deleted
I20260928 15:47:23.643347 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.619785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.13.5.65:42513)
0928 15:47:23.620062 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.620071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.621817 (+ 1746us) server_negotiation.cc:407] Connection header received
0928 15:47:23.621920 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.621929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.622064 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.622196 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:23.625251 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.626895 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.629103 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.629829 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.629931 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.634029 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:23.634052 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:23.634071 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:23.634118 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:23.638750 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.639759 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.639768 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.639772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.639962 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.640559 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.640564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.640568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.641070 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:23.641417 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.642678 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.642931 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 15:47:23.644474 26416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:23.760414 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 26260
W20260928 15:47:23.780822 26208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.5.65:42513 (error 108)
I20260928 15:47:23.780939 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 26425
I20260928 15:47:23.782678 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.782192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57254)
0928 15:47:23.782352 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.782490 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 15:47:23.782924 26208 catalog_manager.cc:4584] TS 318b75522d16428db3f1dadffbb2fd3b (127.13.5.65:42513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1ab94077a1e494fb8a9c5047a6c309f: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
I20260928 15:47:23.798174 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.781728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.13.5.66:43275)
0928 15:47:23.781926 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.781953 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.782006 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:47:23.782240 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.782246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.782335 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.782442 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:23.783943 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.784782 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.786230 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.786865 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.786988 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.789826 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:23.789849 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:23.789891 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:23.789933 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:23.794657 (+ 4724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.795838 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.795848 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.795855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.796070 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.796527 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.796531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.796533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.796913 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:23.797160 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.797536 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.797725 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:47:23.798931 26564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:23.836689 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.836196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57258)
0928 15:47:23.836415 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.836522 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:47:23.838016 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.837529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57274)
0928 15:47:23.837704 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.837836 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 15:47:23.898537 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.897916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57288)
0928 15:47:23.898192 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.898305 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 15:47:23.899680 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.899228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57298)
0928 15:47:23.899411 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.899518 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:47:23.901316 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 26425
I20260928 15:47:23.918637 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 26574
I20260928 15:47:23.932641 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.919469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.13.5.67:44119)
0928 15:47:23.919680 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:23.919686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:23.919707 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:47:23.919905 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:23.919911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:23.919991 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:23.920098 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:23.921417 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.922148 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.923439 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:23.923831 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:23.924032 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:23.926658 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:23.926675 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:23.926717 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:23.926784 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:23.929960 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.930590 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.930597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.930599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.930737 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:23.931082 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:23.931085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:23.931087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:23.931451 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:23.931669 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:23.932092 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 15:47:23.932266 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:47:23.933331 26699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:23.980654 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.980148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57306)
0928 15:47:23.980363 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.980461 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:47:23.981714 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:23.981296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57320)
0928 15:47:23.981529 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:23.981610 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260928 15:47:24.024706 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 26574
I20260928 15:47:24.042675 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 26190
I20260928 15:47:24.044023 26253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:24.159279 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:24.158549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:42513 (local address 127.0.0.1:57328)
0928 15:47:24.158795 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:24.158950 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:42513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:47:24.159682 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 26190
I20260928 15:47:24.186213 13333 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:47:24.521955 13333 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:47:24.522177 13333 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:47:24.868170 13333 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:47:24.868 UTC [25234] LOG: received smart shutdown request
2026-09-28 15:47:24.872 UTC [25234] LOG: background worker "logical replication launcher" (PID 25243) exited with exit code 1
2026-09-28 15:47:24.872 UTC [25238] LOG: shutting down
2026-09-28 15:47:25.064 UTC [25234] LOG: database system is shut down
2026-09-28T15:47:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53942 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:47:25 dist-test-slave-j968 krb5kdc[26772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:47:25 dist-test-slave-j968 krb5kdc[26772](info): set up 2 sockets
Sep 28 15:47:25 dist-test-slave-j968 krb5kdc[26772](info): commencing operation
krb5kdc: starting...
W20260928 15:47:26.338874 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:47:26 dist-test-slave-j968 krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:47:26.588274 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:39655
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:39655,127.13.5.125:39737,127.13.5.124:40451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:26.936429 26785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:26.936765 26785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:26.936932 26785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:26.944664 26785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:26.944768 26785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:26.944868 26785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:26.944973 26785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:26.945056 26785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:26.955277 26785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:39655,127.13.5.125:39737,127.13.5.124:40451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:39655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:26.956777 26785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:26.958289 26785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:26.970623 26790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:26.970700 26791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:26.970731 26793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:26.971617 26785 server_base.cc:1034] running on GCE node
I20260928 15:47:26.972661 26785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:26.974316 26785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:26.975582 26785 hybrid_clock.cc:648] HybridClock initialized: now 1790610446975553 us; error 97 us; skew 500 ppm
I20260928 15:47:26.976059 26785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:26 dist-test-slave-j968 krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610446, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:26.979429 26785 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:47:26.981245 26785 webserver.cc:460] Webserver started at http://127.13.5.126:34483/ using document root <none> and password file <none>
I20260928 15:47:26.982084 26785 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:26.982242 26785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:26.982643 26785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:26.986112 26785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab"
format_stamp: "Formatted at 2026-09-28 15:47:26 on dist-test-slave-j968"
server_key: "9550e075bff3d023f45c502ae35822e2"
server_key_iv: "cc3722796879a2760d71aca749edaf70"
server_key_version: "encryptionkey@0"
I20260928 15:47:26.987006 26785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab"
format_stamp: "Formatted at 2026-09-28 15:47:26 on dist-test-slave-j968"
server_key: "9550e075bff3d023f45c502ae35822e2"
server_key_iv: "cc3722796879a2760d71aca749edaf70"
server_key_version: "encryptionkey@0"
I20260928 15:47:26.992950 26785 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:47:26.996891 26800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:26.998355 26785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:47:26.998557 26785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab"
format_stamp: "Formatted at 2026-09-28 15:47:26 on dist-test-slave-j968"
server_key: "9550e075bff3d023f45c502ae35822e2"
server_key_iv: "cc3722796879a2760d71aca749edaf70"
server_key_version: "encryptionkey@0"
I20260928 15:47:26.998772 26785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:27.019960 26785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:27.024618 26785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:27.024920 26785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:27.044651 26785 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:39655
I20260928 15:47:27.044737 26851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:39655 every 8 connection(s)
I20260928 15:47:27.046087 26785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:47:27.049960 26852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:27.052814 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 26785
I20260928 15:47:27.053042 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:47:27.053900 13333 external_mini_cluster.cc:1428] Setting key bf7aca5f95d9fa09de767a00c97208c8
I20260928 15:47:27.059641 26852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:47:27 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610446, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.125@KRBTEST.COM" created.
I20260928 15:47:27.086215 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.063963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39655 (local address 127.0.0.1:50866)
0928 15:47:27.065176 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.065259 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:27.067417 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:27.068536 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:27.068542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:27.069183 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:27.069591 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:27.069612 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.072132 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.072138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:27.073462 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:27.073477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.075004 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.075010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:27.075135 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:27.075964 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:27.075989 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:27.078261 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:27.081773 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:27.081783 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:27.081793 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:27.082203 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:27.082751 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:27.082754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:27.082771 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:27.082923 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:27.083972 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:27.083978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:27.084188 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 15:47:27.084404 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 15:47:27.085095 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":254,"thread_start_us":468,"threads_started":1}
I20260928 15:47:27.086252 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.063970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.13.5.126:39655)
0928 15:47:27.065890 (+ 1920us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:27.065901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:27.065959 (+ 58us) server_negotiation.cc:407] Connection header received
0928 15:47:27.067857 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:27.067907 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:27.068252 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:27.068480 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:27.069813 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.071891 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.073637 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.074723 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.074954 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:27.078482 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:27.078491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:27.078504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:27.078547 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:27.081618 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:27.082342 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:27.082354 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:27.082360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:27.082594 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:27.083025 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:27.083032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:27.083038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:27.083781 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:27.084019 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:27.084571 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 15:47:27.084982 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1695,"thread_start_us":1479,"threads_started":1}
I20260928 15:47:27.089299 26852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:47:27.090477 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.089987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35426)
0928 15:47:27.090184 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.090321 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 15:47:27.090758 26804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
I20260928 15:47:27.092798 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.092147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35432)
0928 15:47:27.092466 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.092579 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 15:47:27.093048 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:39737: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab.
Entry for principal kudu/127.13.5.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab.
I20260928 15:47:27.114934 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.125:39737
--webserver_interface=127.13.5.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab
--principal=kudu/127.13.5.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:39655,127.13.5.125:39737,127.13.5.124:40451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:47:27.142467 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } attempt: 1
I20260928 15:47:27.143744 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.143264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35440)
0928 15:47:27.143480 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.143599 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 15:47:27.145080 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.144545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35446)
0928 15:47:27.144839 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.144916 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
W20260928 15:47:27.145434 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:39737: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
I20260928 15:47:27.213766 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } attempt: 2
I20260928 15:47:27.215091 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.214596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35448)
0928 15:47:27.214820 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.214943 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:47:27.216598 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.216093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35464)
0928 15:47:27.216259 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.216356 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 15:47:27.216856 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:39737: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
I20260928 15:47:27.308166 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } attempt: 3
I20260928 15:47:27.309473 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.308964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35474)
0928 15:47:27.309188 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.309301 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 15:47:27.310568 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.310198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35490)
0928 15:47:27.310350 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.310426 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:47:27.310792 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:39737: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
W20260928 15:47:27.452117 26860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:27.452409 26860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:27.452572 26860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 15:47:27.454064 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } attempt: 4
I20260928 15:47:27.455273 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.454745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35502)
0928 15:47:27.455051 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.455142 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260928 15:47:27.456348 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.455994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35506)
0928 15:47:27.456141 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.456228 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:47:27.456547 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:39737: Network error: Client connection negotiation failed: client connection to 127.13.5.125:39737: connect: Connection refused (error 111)
W20260928 15:47:27.460194 26860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:27.460314 26860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:27.460408 26860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:27.460507 26860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:27.460598 26860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:27.470758 26860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:39655,127.13.5.125:39737,127.13.5.124:40451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab
--principal=kudu/127.13.5.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.125:39737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.5.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:27.472223 26860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:27.473529 26860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:27.485742 26868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:27.485833 26866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:27.485844 26865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:27.486547 26860 server_base.cc:1034] running on GCE node
I20260928 15:47:27.487279 26860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:27.488801 26860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:27.490016 26860 hybrid_clock.cc:648] HybridClock initialized: now 1790610447489963 us; error 42 us; skew 500 ppm
I20260928 15:47:27.490334 26860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:27 dist-test-slave-j968 krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610447, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:27.493479 26860 init.cc:377] Logged in from keytab as kudu/127.13.5.125@KRBTEST.COM (short username kudu)
I20260928 15:47:27.495049 26860 webserver.cc:460] Webserver started at http://127.13.5.125:38081/ using document root <none> and password file <none>
I20260928 15:47:27.495754 26860 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:27.495929 26860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:27.496250 26860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:27.499310 26860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/data/instance:
uuid: "1bd724f64b264c75b559b1c62d2abb4c"
format_stamp: "Formatted at 2026-09-28 15:47:27 on dist-test-slave-j968"
server_key: "dcc197a4a6d97f23c75b332a3732aafd"
server_key_iv: "b0bf1de3d663419c82b32f5137e53c44"
server_key_version: "encryptionkey@0"
I20260928 15:47:27.500203 26860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/wal/instance:
uuid: "1bd724f64b264c75b559b1c62d2abb4c"
format_stamp: "Formatted at 2026-09-28 15:47:27 on dist-test-slave-j968"
server_key: "dcc197a4a6d97f23c75b332a3732aafd"
server_key_iv: "b0bf1de3d663419c82b32f5137e53c44"
server_key_version: "encryptionkey@0"
I20260928 15:47:27.505741 26860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 15:47:27.509339 26875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:27.510680 26860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:47:27.510851 26860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/wal
uuid: "1bd724f64b264c75b559b1c62d2abb4c"
format_stamp: "Formatted at 2026-09-28 15:47:27 on dist-test-slave-j968"
server_key: "dcc197a4a6d97f23c75b332a3732aafd"
server_key_iv: "b0bf1de3d663419c82b32f5137e53c44"
server_key_version: "encryptionkey@0"
I20260928 15:47:27.511047 26860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:27.532378 26860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:27.536413 26860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:27.536702 26860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:27.555450 26860 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.125:39737
I20260928 15:47:27.555490 26926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.125:39737 every 8 connection(s)
I20260928 15:47:27.556905 26860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
I20260928 15:47:27.558274 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 26860
I20260928 15:47:27.558461 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-1/wal/instance
I20260928 15:47:27.559069 13333 external_mini_cluster.cc:1428] Setting key f6ebbd8e8cf35509ed7119001d1880d7
I20260928 15:47:27.561800 26927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:27.570909 26927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.5.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.124@KRBTEST.COM" created.
Sep 28 15:47:27 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610447, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:27.594206 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.573789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.13.5.126:39655)
0928 15:47:27.574025 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:27.574032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:27.575282 (+ 1250us) server_negotiation.cc:407] Connection header received
0928 15:47:27.577418 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:27.577427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:27.577544 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:27.577678 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:27.579212 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.580400 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.582220 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.582651 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.582751 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:27.586994 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:27.587004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:27.587012 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:27.587062 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:27.590831 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:27.591614 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:27.591623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:27.591629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:27.591791 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:27.592357 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:27.592361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:27.592363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:27.592710 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:27.592821 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:27.593795 (+ 974us) server_negotiation.cc:299] Negotiation successful
0928 15:47:27.593982 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:47:27.595959 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.574247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39655 (local address 127.0.0.1:50876)
0928 15:47:27.575089 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.575141 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:27.577121 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:27.577889 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:27.577897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:27.578419 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:27.579011 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:27.579028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.580575 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.580579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:27.582063 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:27.582095 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.582871 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.582876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:27.583128 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:27.583938 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:27.583975 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:27.586727 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:27.590994 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:27.591002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:27.591017 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:27.591462 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:27.591957 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:27.591964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:27.591970 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:27.592235 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:27.592859 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:27.592871 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:27.593373 (+ 502us) client_negotiation.cc:769] Sending connection context
0928 15:47:27.593697 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 15:47:27.594274 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":370,"threads_started":1}
I20260928 15:47:27.597856 26927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:47:27 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610447, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:47:27.614799 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.598765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35510)
0928 15:47:27.599031 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.599052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:27.599171 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:27.600365 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:27.600368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:27.600758 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:27.600929 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:27.600938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.603173 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.603201 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:27.604180 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:27.604193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.605561 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.605566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:27.605716 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:27.606348 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:27.606366 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:27.608642 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:27.612214 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:27.612218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:27.612221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:27.612606 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:27.613005 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:27.613009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:27.613011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:27.613178 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:27.613908 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:27.613911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:27.614091 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 15:47:27.614216 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 15:47:27.614427 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:47:27.614923 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.599016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.13.5.125:39737)
0928 15:47:27.599773 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:27.599783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:27.599815 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:47:27.599898 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:27.599908 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:27.600158 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:27.600308 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:27.601102 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.602998 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.604326 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.605389 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.605557 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:27.608842 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:27.608887 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:27.608908 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:27.608956 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:27.612076 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:27.612726 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:27.612733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:27.612736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:27.612892 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:27.613278 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:27.613281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:27.613283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:27.613716 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:27.613988 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:27.614369 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 15:47:27.614615 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":402,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab.
Entry for principal kudu/127.13.5.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab.
I20260928 15:47:27.617482 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.124:40451
--webserver_interface=127.13.5.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab
--principal=kudu/127.13.5.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:39655,127.13.5.125:39737,127.13.5.124:40451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:47:27.617939 26927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:47:27.619153 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.618691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53338)
0928 15:47:27.618881 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.619014 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 15:47:27.619542 26878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:27.621281 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.620934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53346)
0928 15:47:27.621101 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.621181 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 15:47:27.621616 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:27.670956 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 1
I20260928 15:47:27.672375 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.671804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53354)
0928 15:47:27.672049 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.672181 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 15:47:27.673501 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.673098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53364)
0928 15:47:27.673260 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.673362 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:47:27.673832 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:27.742111 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 2
I20260928 15:47:27.743419 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.742820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53368)
0928 15:47:27.743079 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.743207 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 15:47:27.744720 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.744296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53378)
0928 15:47:27.744477 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.744576 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:47:27.744987 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:27.756034 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } attempt: 5
Sep 28 15:47:27 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610446, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:47:27.772300 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.756795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35514)
0928 15:47:27.757059 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.757086 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:27.757258 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:27.757680 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:27.757686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:27.758185 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:27.758426 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:27.758439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.759422 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.759427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:27.761116 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:27.761134 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.761946 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.761952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:27.762121 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:27.762829 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:27.762861 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:27.765567 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:27.769809 (+ 4242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:27.769821 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:27.769824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:27.770219 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:27.770645 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:27.770649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:27.770651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:27.770784 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:27.771462 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:27.771467 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:27.771617 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 15:47:27.771735 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:47:27.772035 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 15:47:27.772508 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.756913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.13.5.125:39737)
0928 15:47:27.757093 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:27.757097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:27.757179 (+ 82us) server_negotiation.cc:407] Connection header received
0928 15:47:27.757384 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:27.757388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:27.757517 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:27.757621 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:27.758594 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.759276 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.761290 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:27.761765 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:27.761865 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:27.765832 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:27.765842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:27.765849 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:27.765904 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:27.769582 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:27.770352 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:27.770358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:27.770369 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:27.770530 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:27.770893 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:27.770898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:27.770902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:27.771299 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:27.771497 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:27.771903 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 15:47:27.772166 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:47:27.773669 26852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:47:27.774995 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.774584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53382)
0928 15:47:27.774767 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.774846 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:47:27.776299 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.775777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53390)
0928 15:47:27.776005 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.776133 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 15:47:27.776607 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:27.827925 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 1
I20260928 15:47:27.829201 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.828684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53404)
0928 15:47:27.828908 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.829026 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 15:47:27.830260 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.829907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53414)
0928 15:47:27.830061 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.830142 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:47:27.830514 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:27.836248 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 3
I20260928 15:47:27.837611 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.836999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53426)
0928 15:47:27.837288 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.837401 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 15:47:27.839085 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.838436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53438)
0928 15:47:27.838761 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.838899 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260928 15:47:27.839392 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:27.898777 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 2
I20260928 15:47:27.900092 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.899529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53452)
0928 15:47:27.899770 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.899858 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 15:47:27.901189 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.900848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53454)
0928 15:47:27.901012 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.901077 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 15:47:27.901429 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
W20260928 15:47:27.970723 26934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:27.971037 26934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:27.971251 26934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:27.978817 26934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:27.978927 26934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:27.979015 26934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:27.979131 26934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:27.979207 26934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:27.982636 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 4
I20260928 15:47:27.983848 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.983288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53458)
0928 15:47:27.983551 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.983658 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:47:27.985045 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:27.984590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53474)
0928 15:47:27.984805 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:27.984899 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260928 15:47:27.985291 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:27.990481 26934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:39655,127.13.5.125:39737,127.13.5.124:40451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab
--principal=kudu/127.13.5.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.124:40451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.5.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:27.991919 26934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:27.993348 26934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:28.006193 26940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:28.006265 26939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:28.006265 26942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:28.006879 26934 server_base.cc:1034] running on GCE node
I20260928 15:47:28.007683 26934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:28.007676 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 3
I20260928 15:47:28.008958 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.008347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53484)
0928 15:47:28.008664 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:28.008782 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 15:47:28.009476 26934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:28.010241 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.009797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53486)
0928 15:47:28.010008 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:28.010089 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 15:47:28.010484 26852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:40451: Network error: Client connection negotiation failed: client connection to 127.13.5.124:40451: connect: Connection refused (error 111)
I20260928 15:47:28.010723 26934 hybrid_clock.cc:648] HybridClock initialized: now 1790610448010668 us; error 48 us; skew 500 ppm
I20260928 15:47:28.011078 26934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610448, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:28.014245 26934 init.cc:377] Logged in from keytab as kudu/127.13.5.124@KRBTEST.COM (short username kudu)
I20260928 15:47:28.015743 26934 webserver.cc:460] Webserver started at http://127.13.5.124:44007/ using document root <none> and password file <none>
I20260928 15:47:28.016492 26934 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:28.016593 26934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:28.016912 26934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:28.019846 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/data/instance:
uuid: "7881c8d701d2491f8cfc7395a58f7c60"
format_stamp: "Formatted at 2026-09-28 15:47:28 on dist-test-slave-j968"
server_key: "a07c75db432c4d169e47cfa4d8b64ccf"
server_key_iv: "9ac0fae71bd8999ea7898e6f2d87c201"
server_key_version: "encryptionkey@0"
I20260928 15:47:28.020740 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/wal/instance:
uuid: "7881c8d701d2491f8cfc7395a58f7c60"
format_stamp: "Formatted at 2026-09-28 15:47:28 on dist-test-slave-j968"
server_key: "a07c75db432c4d169e47cfa4d8b64ccf"
server_key_iv: "9ac0fae71bd8999ea7898e6f2d87c201"
server_key_version: "encryptionkey@0"
I20260928 15:47:28.025873 26934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:47:28.029320 26949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:28.030696 26934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:47:28.030881 26934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/wal
uuid: "7881c8d701d2491f8cfc7395a58f7c60"
format_stamp: "Formatted at 2026-09-28 15:47:28 on dist-test-slave-j968"
server_key: "a07c75db432c4d169e47cfa4d8b64ccf"
server_key_iv: "9ac0fae71bd8999ea7898e6f2d87c201"
server_key_version: "encryptionkey@0"
I20260928 15:47:28.031088 26934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:28.052626 26934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:28.056860 26934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:28.057163 26934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:28.076437 26934 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.124:40451
I20260928 15:47:28.076485 27001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.124:40451 every 8 connection(s)
I20260928 15:47:28.077840 26934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
I20260928 15:47:28.082154 27002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:28.082196 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 26934
I20260928 15:47:28.082386 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610238672534-13333-0/minicluster-data/master-2/wal/instance
I20260928 15:47:28.083029 13333 external_mini_cluster.cc:1428] Setting key 8a565ff16906673cb46de58ef29c66e5
I20260928 15:47:28.091424 27002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610448, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:28.107242 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.085047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.13.5.124:40451)
0928 15:47:28.085885 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.085897 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.085935 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:47:28.087583 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.087604 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.088059 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.088465 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.089947 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.091222 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.092730 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.093417 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.093630 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.098345 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.098353 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.098364 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.098407 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.102659 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.103609 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.103617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.103626 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.103852 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.104332 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.104337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.104341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.104947 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.105258 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.105734 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.106189 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":390,"threads_started":1}
I20260928 15:47:28.109786 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.093892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.13.5.126:39655)
0928 15:47:28.094141 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.094150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.094737 (+ 587us) server_negotiation.cc:407] Connection header received
0928 15:47:28.094920 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.094925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.095016 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.095126 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.096321 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.097144 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.098905 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.099683 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.099856 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.104135 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.104140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.104143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.104177 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.106933 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.107729 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.107737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.107741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.107950 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.108332 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.108337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.108341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.108597 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.108762 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.109183 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.109407 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:47:28.109829 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.093856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39655 (local address 127.0.0.1:50880)
0928 15:47:28.094570 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:28.094631 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:28.094793 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:28.095201 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:28.095205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:28.095813 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:28.096159 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.096168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.097310 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.097317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.098720 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.098732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.099989 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.099996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.100144 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.101162 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:28.101181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:28.103956 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:28.107070 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.107083 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.107090 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.107610 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.108066 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.108071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.108075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.108222 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.108728 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:28.108735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:28.108903 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 15:47:28.109094 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 15:47:28.109415 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":645,"threads_started":1}
I20260928 15:47:28.111137 27002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610448, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:28.126360 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.111976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:39737 (local address 127.0.0.1:35524)
0928 15:47:28.112162 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:28.112183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:28.112293 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:28.112721 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:28.112724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:28.113030 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:28.113162 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.113169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.114342 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.114348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.115667 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.115678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.116737 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.116745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.116910 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.117411 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:28.117436 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:28.120219 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:28.123664 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.123670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.123673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.124223 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.124589 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.124595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.124599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.124780 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.125424 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:28.125428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:28.125571 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 15:47:28.125743 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 15:47:28.126051 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 15:47:28.126468 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.112002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.13.5.125:39737)
0928 15:47:28.112193 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.112198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.112250 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:47:28.112491 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.112494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.112555 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.112676 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.113311 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.114196 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.115803 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.116574 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.116674 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.120498 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.120505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.120510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.120553 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.123522 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.124337 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.124341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.124344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.124475 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.124913 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.124920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.124922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.125293 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.125458 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.125846 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.126117 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:47:28.127453 27002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:47:28.127835 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.110485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.13.5.126:39655)
0928 15:47:28.110816 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.110820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.110833 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:47:28.110906 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.110910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.110976 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.111074 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.112679 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.113658 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.115452 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.116177 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.116328 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.121579 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.121587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.121595 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.121654 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.124832 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.125658 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.125663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.125666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.125818 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.126256 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.126261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.126264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.126650 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.126963 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.127286 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.127529 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610448, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:47:28.147512 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.129218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.13.5.124:40451)
0928 15:47:28.129376 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.129381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.129396 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:47:28.129437 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.129442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.129501 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.129621 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.130571 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.131254 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.132813 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.133388 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.133532 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.141943 (+ 8411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.141949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.141954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.141994 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.144899 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.145597 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.145603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.145607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.145720 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.146070 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.146077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.146080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.146359 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.146514 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.146848 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.147082 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:47:28.147612 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.128452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53502)
0928 15:47:28.128650 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:28.128671 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:28.128778 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:28.129635 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:28.129640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:28.130206 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:28.130438 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.130461 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.131409 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.131418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.132654 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.132667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.133654 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.133664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.133904 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.138969 (+ 5065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:28.138998 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:28.141764 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:28.145030 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.145037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.145041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.145483 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.145827 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.145833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.145836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.145969 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.146492 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:28.146497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:28.146641 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 15:47:28.146776 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 15:47:28.147246 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":142}
I20260928 15:47:28.148592 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.129837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.13.5.125:39737)
0928 15:47:28.129996 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.130000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.130146 (+ 146us) server_negotiation.cc:407] Connection header received
0928 15:47:28.130303 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.130306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.130358 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.130430 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.131958 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.133188 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.135218 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.136030 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.136175 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.143156 (+ 6981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.143161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.143164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.143198 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.145787 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.146428 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.146435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.146438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.146580 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.146940 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.146946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.146949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.147217 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.147503 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.148025 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.148264 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:47:28.153913 27002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60: Bootstrap starting.
I20260928 15:47:28.154166 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:47:28.157342 27002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:28.158540 27002 log.cc:824] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:28.166172 27002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60: No bootstrap required, opened a new log
I20260928 15:47:28.178224 27002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
I20260928 15:47:28.178717 27002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:28.178871 27002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7881c8d701d2491f8cfc7395a58f7c60, State: Initialized, Role: FOLLOWER
I20260928 15:47:28.179734 27002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
I20260928 15:47:28.182375 27002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:28.183377 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.159080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.13.5.125:39737)
0928 15:47:28.159251 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.159259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.164630 (+ 5371us) server_negotiation.cc:407] Connection header received
0928 15:47:28.165142 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.165150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.165243 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.165324 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.167634 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.168729 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.170765 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.171409 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.171515 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.175443 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.175450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.175455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.175502 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.179530 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.181003 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.181022 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.181029 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.181193 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.181637 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.181642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.181645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.182020 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.182241 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.182772 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.182998 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:47:28.183715 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.159440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.13.5.126:39655)
0928 15:47:28.159680 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.159687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.162861 (+ 3174us) server_negotiation.cc:407] Connection header received
0928 15:47:28.163547 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.163556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.163665 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.163754 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.166702 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.167828 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.169911 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.170602 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.170714 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.174943 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.174952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.174957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.175020 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.179256 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.180821 (+ 1565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.180827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.180831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.180987 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.181484 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.181489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.181492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.181894 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.182110 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.182803 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.183080 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:47:28.183919 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.159043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.13.5.124:40451)
0928 15:47:28.159211 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.159216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.163187 (+ 3971us) server_negotiation.cc:407] Connection header received
0928 15:47:28.163489 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.163494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.163599 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.163679 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.167106 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.168238 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.170485 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.171225 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.171318 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.175455 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.175462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.175468 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.175515 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.179553 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.180790 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.180796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.180800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.180989 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.181442 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.181447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.181451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.181842 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.182105 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.183263 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.183491 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 15:47:28.184245 27015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.184700 27015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:47:28.187709 26852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 4
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610446, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
W20260928 15:47:28.199061 27026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:47:28.199198 27026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:47:28.204998 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.188377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53520)
0928 15:47:28.188760 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:28.188780 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:28.188910 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:28.189457 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:28.189461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:28.189855 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:28.190020 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.190029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.191422 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.191427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.192626 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.192641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.193525 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.193531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.193723 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.194361 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:28.194380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:28.197223 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:28.201731 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.201737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.201741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.202284 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.202725 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.202733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.202737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.202916 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.203652 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:28.203657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:28.203962 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 15:47:28.204199 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 15:47:28.204598 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260928 15:47:28.205960 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.188715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.13.5.124:40451)
0928 15:47:28.189149 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.189155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.189172 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:47:28.189221 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.189226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.189292 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.189441 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.190165 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.191238 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.192769 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.193367 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.193510 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.197440 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.197452 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.197457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.197494 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.201578 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.202426 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.202432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.202436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.202613 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.203053 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.203060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.203065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.203518 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.203685 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.205302 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.205572 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260928 15:47:28.196611 26934 master_runner.cc:186] Error getting master registration for 127.13.5.126:39655: OK, instance_id { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" instance_seqno: 1790610447039072 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:47:28.208730 26934 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:47:28.211438 26852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: Bootstrap starting.
I20260928 15:47:28.214715 26852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:28.215816 26852 log.cc:824] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:28.220096 26852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: No bootstrap required, opened a new log
I20260928 15:47:28.231441 26852 raft_consensus.cc:348] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
I20260928 15:47:28.231942 26852 raft_consensus.cc:374] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:28.232092 26852 raft_consensus.cc:729] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad89912f6fa4d7aaf008b01d8b6d4ab, State: Initialized, Role: FOLLOWER
I20260928 15:47:28.232828 26852 consensus_queue.cc:260] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
I20260928 15:47:28.233872 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.234229 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:47:28.235318 26852 sys_catalog.cc:564] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:28.239981 27015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:47:28.240309 27015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
I20260928 15:47:28.241987 27015 leader_election.cc:288] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cad89912f6fa4d7aaf008b01d8b6d4ab (127.13.5.126:39655), 1bd724f64b264c75b559b1c62d2abb4c (127.13.5.125:39737)
I20260928 15:47:28.242921 26827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7881c8d701d2491f8cfc7395a58f7c60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" is_pre_election: true
I20260928 15:47:28.243993 26827 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7881c8d701d2491f8cfc7395a58f7c60 in term 0.
I20260928 15:47:28.244901 26950 leader_election.cc:302] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7881c8d701d2491f8cfc7395a58f7c60, cad89912f6fa4d7aaf008b01d8b6d4ab; no voters:
I20260928 15:47:28.245388 27015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:47:28.245541 27015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:47:28.245657 27015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:28.249270 27015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
I20260928 15:47:28.250378 27015 leader_election.cc:288] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [CANDIDATE]: Term 1 election: Requested vote from peers cad89912f6fa4d7aaf008b01d8b6d4ab (127.13.5.126:39655), 1bd724f64b264c75b559b1c62d2abb4c (127.13.5.125:39737)
I20260928 15:47:28.258162 26827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7881c8d701d2491f8cfc7395a58f7c60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab"
I20260928 15:47:28.258400 26827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:28.262136 26827 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7881c8d701d2491f8cfc7395a58f7c60 in term 1.
I20260928 15:47:28.262687 26950 leader_election.cc:302] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7881c8d701d2491f8cfc7395a58f7c60, cad89912f6fa4d7aaf008b01d8b6d4ab; no voters:
I20260928 15:47:28.263115 27015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:47:28.249316 26785 master_runner.cc:186] Error getting master registration for 127.13.5.125:39737: OK, instance_id { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" instance_seqno: 1790610447550055 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:47:28.264601 26785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 15:47:28.266369 27040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:47:28.266492 27040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:47:28.266909 27015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [term 1 LEADER]: Becoming Leader. State: Replica: 7881c8d701d2491f8cfc7395a58f7c60, State: Running, Role: LEADER
I20260928 15:47:28.267511 27015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
W20260928 15:47:28.278148 26806 tablet.cc:2369] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:47:28.280797 27042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7881c8d701d2491f8cfc7395a58f7c60. Latest consensus state: current_term: 1 leader_uuid: "7881c8d701d2491f8cfc7395a58f7c60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.281262 27042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:28.282348 27044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:47:28.284482 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } attempt: 5
I20260928 15:47:28.284698 27044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 15:47:28 dist-test-slave-j968 krb5kdc[26772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610447, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
I20260928 15:47:28.297245 26827 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [term 1 FOLLOWER]: Refusing update from remote peer 7881c8d701d2491f8cfc7395a58f7c60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:47:28.298245 27042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:47:28.305416 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.285439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.13.5.124:40451)
0928 15:47:28.285617 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:28.285625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:28.285647 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:47:28.285706 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:28.285711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:28.285786 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:28.285897 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:28.286875 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.288117 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.289879 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.290590 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.290733 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.295153 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:28.295160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:28.295165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:28.295205 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:28.299490 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.300846 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.300852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.300856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.301118 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:28.301868 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:28.301873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:28.301877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:28.302592 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:28.302810 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:28.303823 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0928 15:47:28.304317 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:47:28.306588 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:28.285086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:40451 (local address 127.0.0.1:53524)
0928 15:47:28.285286 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:28.285308 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:28.285444 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:28.285926 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:28.285931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:28.286500 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:28.286722 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.286735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.288266 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.288272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.289701 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:28.289718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:28.290956 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:28.290964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:28.291246 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:28.291972 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:28.291997 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:28.294917 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:28.299742 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.299748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.299752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.300646 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.301399 (+ 753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:28.301428 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:28.301432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:28.301693 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:28.302981 (+ 1288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:28.302986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:28.303334 (+ 348us) client_negotiation.cc:769] Sending connection context
0928 15:47:28.304701 (+ 1367us) client_negotiation.cc:240] Negotiation successful
0928 15:47:28.305823 (+ 1122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:47:28.309315 26955 tablet.cc:2369] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:47:28.314519 26927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c: Bootstrap starting.
I20260928 15:47:28.319034 26927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:28.320446 26927 log.cc:824] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:28.328382 26927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c: No bootstrap required, opened a new log
I20260928 15:47:28.332968 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7881c8d701d2491f8cfc7395a58f7c60. Latest consensus state: current_term: 1 leader_uuid: "7881c8d701d2491f8cfc7395a58f7c60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.333310 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:47:28.337239 27045 mvcc.cc:205] Tried to move back new op lower bound from 7334340396213714944 to 7334340396109025280. Current Snapshot: MvccSnapshot[applied={T|T < 7334340396213714944}]
I20260928 15:47:28.338709 27046 mvcc.cc:205] Tried to move back new op lower bound from 7334340396213714944 to 7334340396109025280. Current Snapshot: MvccSnapshot[applied={T|T < 7334340396213714944}]
I20260928 15:47:28.339301 27042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7881c8d701d2491f8cfc7395a58f7c60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.339779 27042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:28.340539 27029 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7881c8d701d2491f8cfc7395a58f7c60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.340793 27029 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:47:28.347308 27044 catalog_manager.cc:1347] Generated new cluster ID: 161ef82f7ee54377b1aaeec33bfba07c
I20260928 15:47:28.347421 27044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:47:28.349138 26927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
I20260928 15:47:28.349509 26927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:28.349606 26927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd724f64b264c75b559b1c62d2abb4c, State: Initialized, Role: FOLLOWER
I20260928 15:47:28.350292 26927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } }
I20260928 15:47:28.357199 27051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.357631 27051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:47:28.359558 26927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:28.359620 26902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7881c8d701d2491f8cfc7395a58f7c60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd724f64b264c75b559b1c62d2abb4c" is_pre_election: true
I20260928 15:47:28.360251 26902 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7881c8d701d2491f8cfc7395a58f7c60 in term 0.
I20260928 15:47:28.360795 26901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7881c8d701d2491f8cfc7395a58f7c60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd724f64b264c75b559b1c62d2abb4c"
I20260928 15:47:28.361021 26901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:28.365566 26901 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7881c8d701d2491f8cfc7395a58f7c60 in term 1.
I20260928 15:47:28.366660 26900 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [term 1 FOLLOWER]: Refusing update from remote peer 7881c8d701d2491f8cfc7395a58f7c60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:47:28.367688 27042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:47:28.401805 27044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:47:28.404505 27044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:47:28.411497 27057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7881c8d701d2491f8cfc7395a58f7c60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.411937 27057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:47:28.412345 27042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7881c8d701d2491f8cfc7395a58f7c60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.412618 27042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:28.412849 27051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7881c8d701d2491f8cfc7395a58f7c60. Latest consensus state: current_term: 1 leader_uuid: "7881c8d701d2491f8cfc7395a58f7c60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cad89912f6fa4d7aaf008b01d8b6d4ab" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39655 } } peers { permanent_uuid: "1bd724f64b264c75b559b1c62d2abb4c" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 39737 } } peers { permanent_uuid: "7881c8d701d2491f8cfc7395a58f7c60" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 40451 } } }
I20260928 15:47:28.413180 27051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:47:28.443800 27044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7881c8d701d2491f8cfc7395a58f7c60: Generated new TSK 0
I20260928 15:47:28.445055 27044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:47:28.497742 27068 catalog_manager.cc:1259] Loaded cluster ID: 161ef82f7ee54377b1aaeec33bfba07c
I20260928 15:47:28.497881 27068 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c: loading cluster ID for follower catalog manager: success
I20260928 15:47:28.502038 27068 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c: acquiring CA information for follower catalog manager: success
I20260928 15:47:28.504792 27068 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1bd724f64b264c75b559b1c62d2abb4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:47:29.268743 27040 catalog_manager.cc:1259] Loaded cluster ID: 161ef82f7ee54377b1aaeec33bfba07c
I20260928 15:47:29.268850 27040 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: loading cluster ID for follower catalog manager: success
I20260928 15:47:29.271250 27040 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: acquiring CA information for follower catalog manager: success
I20260928 15:47:29.273016 27040 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cad89912f6fa4d7aaf008b01d8b6d4ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:47:29.595219 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 26785
I20260928 15:47:29.596861 26847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:29.711037 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 26785
I20260928 15:47:29.734210 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 26860
I20260928 15:47:29.735491 26922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:29.846755 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 26860
I20260928 15:47:29.868376 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 26934
I20260928 15:47:29.869686 26997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:29.980512 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 26934
2026-09-28T15:47:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4790 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:47:30 dist-test-slave-j968 krb5kdc[27079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:47:30 dist-test-slave-j968 krb5kdc[27079](info): set up 2 sockets
Sep 28 15:47:30 dist-test-slave-j968 krb5kdc[27079](info): commencing operation
krb5kdc: starting...
W20260928 15:47:31.118927 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:47:31 dist-test-slave-j968 krb5kdc[27079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:47:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:47:31.373885 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36339
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:36339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:31.717831 27092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:31.718195 27092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:31.718360 27092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:31.726315 27092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:31.726477 27092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:31.726599 27092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:31.726712 27092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:31.726783 27092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:31.738628 27092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:36339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:31.740177 27092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:31.741837 27092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:31.755086 27097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:31.755105 27100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:31.755163 27098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:31.755931 27092 server_base.cc:1034] running on GCE node
I20260928 15:47:31.756837 27092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:31.758407 27092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:31.759639 27092 hybrid_clock.cc:648] HybridClock initialized: now 1790610451759580 us; error 43 us; skew 500 ppm
I20260928 15:47:31.760043 27092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:31 dist-test-slave-j968 krb5kdc[27079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610451, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:31.764006 27092 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:47:31.765733 27092 webserver.cc:460] Webserver started at http://127.13.5.126:43671/ using document root <none> and password file <none>
I20260928 15:47:31.766476 27092 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:31.766607 27092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:31.766979 27092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:31.770186 27092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "eb64c2c1dbf74fc9af4179b775dc05fc"
format_stamp: "Formatted at 2026-09-28 15:47:31 on dist-test-slave-j968"
server_key: "ed494085486403152032566bd00c6abb"
server_key_iv: "0447df47af36539dcd2a94553e91be89"
server_key_version: "encryptionkey@0"
I20260928 15:47:31.770983 27092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "eb64c2c1dbf74fc9af4179b775dc05fc"
format_stamp: "Formatted at 2026-09-28 15:47:31 on dist-test-slave-j968"
server_key: "ed494085486403152032566bd00c6abb"
server_key_iv: "0447df47af36539dcd2a94553e91be89"
server_key_version: "encryptionkey@0"
I20260928 15:47:31.777463 27092 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:47:31.781219 27107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:31.782765 27092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:47:31.782980 27092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "eb64c2c1dbf74fc9af4179b775dc05fc"
format_stamp: "Formatted at 2026-09-28 15:47:31 on dist-test-slave-j968"
server_key: "ed494085486403152032566bd00c6abb"
server_key_iv: "0447df47af36539dcd2a94553e91be89"
server_key_version: "encryptionkey@0"
I20260928 15:47:31.783205 27092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:31.798521 27092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:31.803244 27092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:31.803576 27092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:31.823385 27092 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:36339
I20260928 15:47:31.823408 27158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:36339 every 8 connection(s)
I20260928 15:47:31.825016 27092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:47:31.828507 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 27092
I20260928 15:47:31.828740 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:47:31.829303 27159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:31.829535 13333 external_mini_cluster.cc:1428] Setting key c7636aaf624e293f0a187c41fa264091
I20260928 15:47:31.844949 27159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc: Bootstrap starting.
Sep 28 15:47:31 dist-test-slave-j968 krb5kdc[27079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:31.849023 27159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:31.850203 27159 log.cc:824] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:31.853519 27159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc: No bootstrap required, opened a new log
I20260928 15:47:31.855275 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:31.831430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.13.5.126:36339)
0928 15:47:31.832293 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:31.832305 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:31.832352 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:47:31.834329 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:31.834354 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:31.834814 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:31.835195 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:31.837196 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:31.838548 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:31.840524 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:31.841303 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:31.841461 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:31.846725 (+ 5264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:31.846737 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:31.846758 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:31.846814 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:31.850804 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:31.851643 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:31.851650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:31.851657 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:31.851824 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:31.852320 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:31.852325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:31.852328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:31.852893 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:31.853198 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:31.853692 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 15:47:31.854252 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":397,"threads_started":1}
I20260928 15:47:31.859434 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:47:31.862061 27159 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb64c2c1dbf74fc9af4179b775dc05fc" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36339 } }
I20260928 15:47:31.862485 27159 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:31.862607 27159 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb64c2c1dbf74fc9af4179b775dc05fc, State: Initialized, Role: FOLLOWER
I20260928 15:47:31.863210 27159 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb64c2c1dbf74fc9af4179b775dc05fc" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36339 } }
I20260928 15:47:31.863425 27159 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:47:31.863533 27159 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:47:31.863662 27159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:31.865846 27159 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb64c2c1dbf74fc9af4179b775dc05fc" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36339 } }
I20260928 15:47:31.866319 27159 leader_election.cc:302] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb64c2c1dbf74fc9af4179b775dc05fc; no voters:
I20260928 15:47:31.866845 27159 leader_election.cc:288] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:47:31.867076 27164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:47:31.867808 27164 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [term 1 LEADER]: Becoming Leader. State: Replica: eb64c2c1dbf74fc9af4179b775dc05fc, State: Running, Role: LEADER
I20260928 15:47:31.868389 27164 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb64c2c1dbf74fc9af4179b775dc05fc" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36339 } }
I20260928 15:47:31.869138 27159 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:31.871409 27166 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb64c2c1dbf74fc9af4179b775dc05fc. Latest consensus state: current_term: 1 leader_uuid: "eb64c2c1dbf74fc9af4179b775dc05fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb64c2c1dbf74fc9af4179b775dc05fc" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36339 } } }
I20260928 15:47:31.871891 27166 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:31.872740 27165 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb64c2c1dbf74fc9af4179b775dc05fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb64c2c1dbf74fc9af4179b775dc05fc" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36339 } } }
I20260928 15:47:31.873421 27165 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:31.874169 27171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:47:31.878971 27171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:47:31.888064 27171 catalog_manager.cc:1347] Generated new cluster ID: a150eba4247f42c0a85f5e829f929e45
I20260928 15:47:31.888234 27171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:47:31.903452 27171 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:47:31.905577 27171 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:47:31.924924 27171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb64c2c1dbf74fc9af4179b775dc05fc: Generated new TSK 0
I20260928 15:47:31.926129 27171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:47:31.972649 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 27092
I20260928 15:47:31.974138 27154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:32.073876 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 27092
2026-09-28T15:47:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2088 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:47:32 dist-test-slave-j968 krb5kdc[27187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:47:32 dist-test-slave-j968 krb5kdc[27187](info): set up 2 sockets
Sep 28 15:47:32 dist-test-slave-j968 krb5kdc[27187](info): commencing operation
krb5kdc: starting...
W20260928 15:47:33.203243 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:47:33 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:47:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:47:33.453676 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:38275
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:33.812106 27200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:33.812502 27200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:33.812685 27200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:33.821831 27200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:33.821949 27200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:33.822001 27200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:33.822068 27200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:33.822116 27200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:33.833412 27200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:38275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:33.834807 27200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:33.836333 27200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:33.849818 27205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:33.849807 27206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:33.849885 27208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:33.856195 27200 server_base.cc:1034] running on GCE node
I20260928 15:47:33.857366 27200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:33.859040 27200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:33.860288 27200 hybrid_clock.cc:648] HybridClock initialized: now 1790610453860223 us; error 58 us; skew 500 ppm
I20260928 15:47:33.860682 27200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:33 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610453, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:33.870780 27200 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:47:33.872857 27200 webserver.cc:460] Webserver started at http://127.13.5.126:35313/ using document root <none> and password file <none>
I20260928 15:47:33.873701 27200 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:33.873862 27200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:33.874213 27200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:33.877440 27200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "af039d6ff1864cf793c47ee56f4a2de0"
format_stamp: "Formatted at 2026-09-28 15:47:33 on dist-test-slave-j968"
server_key: "2cc946721e08c351a4e12d7b196614c2"
server_key_iv: "ff4d7323cb298201e3c67ac3279ddb8e"
server_key_version: "encryptionkey@0"
I20260928 15:47:33.878254 27200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "af039d6ff1864cf793c47ee56f4a2de0"
format_stamp: "Formatted at 2026-09-28 15:47:33 on dist-test-slave-j968"
server_key: "2cc946721e08c351a4e12d7b196614c2"
server_key_iv: "ff4d7323cb298201e3c67ac3279ddb8e"
server_key_version: "encryptionkey@0"
I20260928 15:47:33.883841 27200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:47:33.887621 27215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:33.889318 27200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:47:33.889540 27200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "af039d6ff1864cf793c47ee56f4a2de0"
format_stamp: "Formatted at 2026-09-28 15:47:33 on dist-test-slave-j968"
server_key: "2cc946721e08c351a4e12d7b196614c2"
server_key_iv: "ff4d7323cb298201e3c67ac3279ddb8e"
server_key_version: "encryptionkey@0"
I20260928 15:47:33.889765 27200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:33.913980 27200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:33.918691 27200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:33.919003 27200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:33.939684 27200 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:38275
I20260928 15:47:33.939708 27266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:38275 every 8 connection(s)
I20260928 15:47:33.941274 27200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:47:33.945209 27267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:33.949579 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 27200
I20260928 15:47:33.949806 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:47:33.950678 13333 external_mini_cluster.cc:1428] Setting key 06e36c583422e97b8ecb0751334c3ee8
I20260928 15:47:33.957852 27267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Bootstrap starting.
I20260928 15:47:33.961102 27267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:33.962309 27267 log.cc:824] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:33.965329 27267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: No bootstrap required, opened a new log
Sep 28 15:47:33 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:33.975494 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:33.953250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.13.5.126:38275)
0928 15:47:33.954092 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:33.954105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:33.954149 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:47:33.956602 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:33.956625 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:33.957049 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:33.957449 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:33.958937 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:33.960413 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:33.962003 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:33.962746 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:33.963005 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:33.967466 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:33.967477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:33.967492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:33.967535 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:33.970903 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:33.971810 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:33.971816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:33.971822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:33.972063 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:33.972536 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:33.972539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:33.972541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:33.973133 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:33.973366 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:33.973885 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 15:47:33.974381 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":372,"threads_started":1}
I20260928 15:47:33.975267 27267 raft_consensus.cc:348] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:33.975713 27267 raft_consensus.cc:374] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:33.975842 27267 raft_consensus.cc:729] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af039d6ff1864cf793c47ee56f4a2de0, State: Initialized, Role: FOLLOWER
I20260928 15:47:33.976492 27267 consensus_queue.cc:260] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:33.976770 27267 raft_consensus.cc:388] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:47:33.976922 27267 raft_consensus.cc:482] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:47:33.977118 27267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:33.979494 27267 raft_consensus.cc:504] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:33.980089 27267 leader_election.cc:302] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af039d6ff1864cf793c47ee56f4a2de0; no voters:
I20260928 15:47:33.980783 27267 leader_election.cc:288] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:47:33.980927 27273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:47:33.981699 27273 raft_consensus.cc:686] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 1 LEADER]: Becoming Leader. State: Replica: af039d6ff1864cf793c47ee56f4a2de0, State: Running, Role: LEADER
I20260928 15:47:33.982266 27273 consensus_queue.cc:237] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:33.982972 27267 sys_catalog.cc:564] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:33.985383 27275 sys_catalog.cc:455] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af039d6ff1864cf793c47ee56f4a2de0. Latest consensus state: current_term: 1 leader_uuid: "af039d6ff1864cf793c47ee56f4a2de0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } } }
I20260928 15:47:33.985819 27275 sys_catalog.cc:458] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:33.986363 27274 sys_catalog.cc:455] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af039d6ff1864cf793c47ee56f4a2de0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } } }
I20260928 15:47:33.986795 27274 sys_catalog.cc:458] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:33.993366 27280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:47:33.998538 27280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:47:34.008541 27280 catalog_manager.cc:1347] Generated new cluster ID: 7a991cbddfe54bd7a444607435c87f94
I20260928 15:47:34.008694 27280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:47:34.016281 27280 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:47:34.017675 27280 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:47:34.026635 27280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Generated new TSK 0
I20260928 15:47:34.027743 27280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:47:34.092495 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38275
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:34.438979 27295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:34.439337 27295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:34.439502 27295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:34.447111 27295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:34.447273 27295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:34.447481 27295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:47:34.457541 27295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:34.458920 27295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:34.460404 27295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:34.474174 27300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:34.474213 27301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:34.474253 27303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:34.476205 27295 server_base.cc:1034] running on GCE node
I20260928 15:47:34.477010 27295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:34.478104 27295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:34.479328 27295 hybrid_clock.cc:648] HybridClock initialized: now 1790610454479266 us; error 59 us; skew 500 ppm
I20260928 15:47:34.479671 27295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:34 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610454, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:34.483019 27295 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:47:34.484678 27295 webserver.cc:460] Webserver started at http://127.13.5.65:35959/ using document root <none> and password file <none>
I20260928 15:47:34.485371 27295 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:34.485517 27295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:34.485834 27295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:34.488862 27295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "c1e7954c1adc4d79930d0ca433bb9853"
format_stamp: "Formatted at 2026-09-28 15:47:34 on dist-test-slave-j968"
server_key: "b48b8e01f08d5d8ba4e290932deab758"
server_key_iv: "01502c140bfda56f1bcc1d9be4ce0d68"
server_key_version: "encryptionkey@0"
I20260928 15:47:34.489655 27295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "c1e7954c1adc4d79930d0ca433bb9853"
format_stamp: "Formatted at 2026-09-28 15:47:34 on dist-test-slave-j968"
server_key: "b48b8e01f08d5d8ba4e290932deab758"
server_key_iv: "01502c140bfda56f1bcc1d9be4ce0d68"
server_key_version: "encryptionkey@0"
I20260928 15:47:34.495188 27295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:47:34.498710 27310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:34.500137 27295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:47:34.500305 27295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "c1e7954c1adc4d79930d0ca433bb9853"
format_stamp: "Formatted at 2026-09-28 15:47:34 on dist-test-slave-j968"
server_key: "b48b8e01f08d5d8ba4e290932deab758"
server_key_iv: "01502c140bfda56f1bcc1d9be4ce0d68"
server_key_version: "encryptionkey@0"
I20260928 15:47:34.500519 27295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:34.527606 27295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:34.532511 27295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:34.532857 27295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:34.533949 27295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:34.535495 27295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:34.535614 27295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:34.535784 27295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:34.535866 27295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:34.572368 27295 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43301
I20260928 15:47:34.572538 27423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43301 every 8 connection(s)
I20260928 15:47:34.573760 27295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:47:34.580319 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 27295
I20260928 15:47:34.580643 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:47:34.581287 13333 external_mini_cluster.cc:1428] Setting key 9ea1a42bdaa777a18ec8bab907c09d72
Sep 28 15:47:34 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610454, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:34.595948 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:34.576581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:37809 (local address 127.13.5.126:38275)
0928 15:47:34.577047 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:34.577052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:34.578123 (+ 1071us) server_negotiation.cc:407] Connection header received
0928 15:47:34.580036 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:34.580044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:34.580143 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:34.580282 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:34.582061 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:34.583150 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:34.584876 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:34.585287 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:34.585438 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:34.589529 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:34.589535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:34.589538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:34.589573 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:34.592343 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:34.593178 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:34.593190 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:34.593192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:34.593315 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:34.593938 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:34.593941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:34.593943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:34.594282 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:34.594471 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:34.595432 (+ 961us) server_negotiation.cc:299] Negotiation successful
0928 15:47:34.595615 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260928 15:47:34.597561 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:34.577168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:37809)
0928 15:47:34.577952 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:34.577999 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:34.579676 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:34.580595 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:34.580604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:34.581168 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:34.581869 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:34.581896 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:34.583332 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:34.583342 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:34.584710 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:34.584723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:34.585545 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:34.585554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:34.585798 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:34.586635 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:34.586659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:34.589256 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:34.592502 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:34.592512 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:34.592533 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:34.593050 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:34.593475 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:34.593483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:34.593490 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:34.593759 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:34.594475 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:34.594486 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:34.594968 (+ 482us) client_negotiation.cc:769] Sending connection context
0928 15:47:34.595324 (+ 356us) client_negotiation.cc:240] Negotiation successful
0928 15:47:34.595968 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":325,"threads_started":1}
I20260928 15:47:34.599588 27424 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38275
I20260928 15:47:34.600010 27424 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:34.600832 27424 heartbeater.cc:511] Master 127.13.5.126:38275 requested a full tablet report, sending...
I20260928 15:47:34.603341 27232 ts_manager.cc:194] Registered new tserver with Master: c1e7954c1adc4d79930d0ca433bb9853 (127.13.5.65:43301)
I20260928 15:47:34.605057 27232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:37809
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:47:34.644455 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38275
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:34.987192 27430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:34.987519 27430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:34.987704 27430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:34.995177 27430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:34.995330 27430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:34.995517 27430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:47:35.006289 27430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:35.007699 27430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:35.009151 27430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:35.023204 27438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:35.023147 27435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:35.023262 27436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:35.028764 27430 server_base.cc:1034] running on GCE node
I20260928 15:47:35.029524 27430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:35.030740 27430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:35.032049 27430 hybrid_clock.cc:648] HybridClock initialized: now 1790610455032002 us; error 74 us; skew 500 ppm
I20260928 15:47:35.032541 27430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:35 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610455, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:35.036345 27430 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:47:35.038004 27430 webserver.cc:460] Webserver started at http://127.13.5.66:46423/ using document root <none> and password file <none>
I20260928 15:47:35.038756 27430 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:35.038915 27430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:35.039242 27430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:35.042363 27430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "010c6024902d491a9081b4ffd8998727"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "b782ab04666a10c0d3eed00c7491b83c"
server_key_iv: "7abdd0fefc0debc89f7a135c3083e6d8"
server_key_version: "encryptionkey@0"
I20260928 15:47:35.043190 27430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "010c6024902d491a9081b4ffd8998727"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "b782ab04666a10c0d3eed00c7491b83c"
server_key_iv: "7abdd0fefc0debc89f7a135c3083e6d8"
server_key_version: "encryptionkey@0"
I20260928 15:47:35.049100 27430 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 15:47:35.053165 27446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:35.054773 27430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:47:35.055006 27430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "010c6024902d491a9081b4ffd8998727"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "b782ab04666a10c0d3eed00c7491b83c"
server_key_iv: "7abdd0fefc0debc89f7a135c3083e6d8"
server_key_version: "encryptionkey@0"
I20260928 15:47:35.055253 27430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:35.081553 27430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:35.086601 27430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:35.086952 27430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:35.088199 27430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:35.089725 27430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:35.089846 27430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:35.090041 27430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:35.090131 27430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:35.129309 27430 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43651
I20260928 15:47:35.129470 27558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43651 every 8 connection(s)
I20260928 15:47:35.130818 27430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:47:35.139603 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 27430
I20260928 15:47:35.139796 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:47:35.140280 13333 external_mini_cluster.cc:1428] Setting key 9da8812e4c403aeaf9c4fa265ebb9216
Sep 28 15:47:35 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610455, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:35.154858 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.133994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:37677 (local address 127.13.5.126:38275)
0928 15:47:35.134635 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.134639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.135632 (+ 993us) server_negotiation.cc:407] Connection header received
0928 15:47:35.137593 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.137600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.137700 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.137823 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:35.139401 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.140536 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.142221 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.142823 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.142965 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.147011 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:35.147021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:35.147029 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:35.147074 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:35.150868 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:35.152034 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:35.152039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:35.152042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:35.152200 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:35.152870 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:35.152874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:35.152877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:35.153221 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:35.153348 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.154354 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.154555 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":411,"threads_started":1}
I20260928 15:47:35.156586 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.134582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:37677)
0928 15:47:35.135369 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:35.135493 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:35.137307 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:35.138061 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:35.138070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:35.138594 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:35.139169 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.139215 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.140701 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.140706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.142063 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.142075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.143100 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.143106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.143326 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.144113 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:35.144137 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:35.146748 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:35.151040 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:35.151055 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:35.151077 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:35.151858 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:35.152332 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:35.152341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:35.152348 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:35.152719 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:35.153363 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:35.153376 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:35.153902 (+ 526us) client_negotiation.cc:769] Sending connection context
0928 15:47:35.154235 (+ 333us) client_negotiation.cc:240] Negotiation successful
0928 15:47:35.154919 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":318,"threads_started":1}
I20260928 15:47:35.158283 27559 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38275
I20260928 15:47:35.158674 27559 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:35.159659 27559 heartbeater.cc:511] Master 127.13.5.126:38275 requested a full tablet report, sending...
I20260928 15:47:35.161536 27232 ts_manager.cc:194] Registered new tserver with Master: 010c6024902d491a9081b4ffd8998727 (127.13.5.66:43651)
I20260928 15:47:35.162483 27232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:37677
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:47:35.199981 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38275
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:35.538384 27565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:35.538671 27565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:35.538764 27565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:35.546254 27565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:35.546378 27565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:35.546492 27565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:47:35.556424 27565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:35.557693 27565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:35.559019 27565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:35.573491 27570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:35.573406 27573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:35.577708 27565 server_base.cc:1034] running on GCE node
W20260928 15:47:35.578594 27571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:35.579340 27565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:35.580408 27565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:35.581647 27565 hybrid_clock.cc:648] HybridClock initialized: now 1790610455581598 us; error 39 us; skew 500 ppm
I20260928 15:47:35.582003 27565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:35 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610455, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:35.585659 27565 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:47:35.587318 27565 webserver.cc:460] Webserver started at http://127.13.5.67:41369/ using document root <none> and password file <none>
I20260928 15:47:35.588150 27565 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:35.588295 27565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:35.588617 27565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:35.591734 27565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "f4107993bff64509b58337ead5767860"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "ef6b0cea4496ab09574914e1d524434e"
server_key_iv: "09eae94ddb7be650d9baac3fb2950ca4"
server_key_version: "encryptionkey@0"
I20260928 15:47:35.592670 27565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "f4107993bff64509b58337ead5767860"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "ef6b0cea4496ab09574914e1d524434e"
server_key_iv: "09eae94ddb7be650d9baac3fb2950ca4"
server_key_version: "encryptionkey@0"
I20260928 15:47:35.598414 27565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 15:47:35.602284 27580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:35.604060 27565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:47:35.604262 27565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "f4107993bff64509b58337ead5767860"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "ef6b0cea4496ab09574914e1d524434e"
server_key_iv: "09eae94ddb7be650d9baac3fb2950ca4"
server_key_version: "encryptionkey@0"
I20260928 15:47:35.604475 27565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:35.608500 27424 heartbeater.cc:503] Master 127.13.5.126:38275 was elected leader, sending a full tablet report...
I20260928 15:47:35.623812 27565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:35.628334 27565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:35.628674 27565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:35.629695 27565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:35.631280 27565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:35.631440 27565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:35.631639 27565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:35.631726 27565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:35.667459 27565 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:36713
I20260928 15:47:35.667562 27693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:36713 every 8 connection(s)
I20260928 15:47:35.668952 27565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:47:35.674949 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 27565
I20260928 15:47:35.675141 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:47:35.675711 13333 external_mini_cluster.cc:1428] Setting key c54126c06ebc81237d633ecbff0e6964
Sep 28 15:47:35 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610455, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:35.692581 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.672019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:46583 (local address 127.13.5.126:38275)
0928 15:47:35.672518 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.672523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.673583 (+ 1060us) server_negotiation.cc:407] Connection header received
0928 15:47:35.676215 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.676219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.676295 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.676435 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:35.678629 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.679636 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.681813 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.682214 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.682317 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.686359 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:35.686365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:35.686368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:35.686404 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:35.689006 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:35.689827 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:35.689831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:35.689834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:35.690047 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:35.690565 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:35.690570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:35.690572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:35.690998 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:35.691123 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.692111 (+ 988us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.692311 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":266,"threads_started":1}
I20260928 15:47:35.693873 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.672539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:46583)
0928 15:47:35.673337 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:35.673378 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:35.675792 (+ 2414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:35.676730 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:35.676744 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:35.677575 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:35.678395 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.678416 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.679799 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.679804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.681651 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.681670 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.682480 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.682486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.682776 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.683468 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:35.683499 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:35.686127 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:35.689142 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:35.689149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:35.689164 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:35.689694 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:35.690172 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:35.690176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:35.690179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:35.690434 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:35.691135 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:35.691143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:35.691611 (+ 468us) client_negotiation.cc:769] Sending connection context
0928 15:47:35.691952 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 15:47:35.692505 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":321,"threads_started":1}
I20260928 15:47:35.695405 27694 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38275
I20260928 15:47:35.695948 27694 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:35.696823 27694 heartbeater.cc:511] Master 127.13.5.126:38275 requested a full tablet report, sending...
I20260928 15:47:35.698601 27232 ts_manager.cc:194] Registered new tserver with Master: f4107993bff64509b58337ead5767860 (127.13.5.67:36713)
I20260928 15:47:35.699467 27232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:46583
I20260928 15:47:35.706257 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:47:35.727562 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.713693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.13.5.126:38275)
0928 15:47:35.713959 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.713965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.714433 (+ 468us) server_negotiation.cc:407] Connection header received
0928 15:47:35.714581 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.714585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.714658 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.714750 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:35.716547 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.717316 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.718915 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.719357 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.719475 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.722274 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:35.722280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:35.722283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:35.722317 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:35.725022 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:35.725799 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:35.725804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:35.725806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:35.725930 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:35.726331 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:35.726334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:35.726336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:35.726674 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:35.726858 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.727161 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.727331 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 15:47:35.732111 27232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:47:35.739033 27232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:47:35.764954 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.752259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.13.5.66:43651)
0928 15:47:35.754629 (+ 2370us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.754642 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.754677 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:47:35.754781 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.754790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.755085 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.756365 (+ 1280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:35.756691 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.758740 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.761788 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.763366 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.763505 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.763729 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.764396 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.764599 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2147,"thread_start_us":1998,"threads_started":1}
I20260928 15:47:35.765059 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.752782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:45816)
0928 15:47:35.753766 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:35.753788 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:35.753900 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:35.755819 (+ 1919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:35.755827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:35.755904 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:35.756477 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.756493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.760030 (+ 3537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.760036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.762160 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.762174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.764089 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.764096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.764330 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.764364 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:47:35.764456 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:47:35.764596 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":2,"thread_start_us":1221,"threads_started":1}
I20260928 15:47:35.765241 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.753499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.13.5.65:43301)
0928 15:47:35.755078 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.755091 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.755127 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:47:35.755247 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.755258 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.755556 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.756746 (+ 1190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:35.757042 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.759845 (+ 2803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.762350 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.763824 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.764013 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.764260 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.764737 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.764912 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":1183,"threads_started":1}
I20260928 15:47:35.765714 27707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.752075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43651 (local address 127.0.0.1:34672)
0928 15:47:35.752879 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:35.752933 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:35.753116 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:35.755380 (+ 2264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:35.755391 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:35.755448 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:35.756075 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.756092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.758954 (+ 2862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.758963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.761588 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.761606 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.763582 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.763589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.763936 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.764011 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 15:47:35.764142 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:47:35.764372 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":27,"thread_start_us":320,"threads_started":1}
I20260928 15:47:35.770056 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.753474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:36713 (local address 127.0.0.1:55718)
0928 15:47:35.759067 (+ 5593us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:35.759084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:35.759211 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:35.761087 (+ 1876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:35.761092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:35.761121 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:35.761662 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.761675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.763586 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.763589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.766756 (+ 3167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.766770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.769149 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.769156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.769332 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.769369 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:47:35.769470 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:47:35.769627 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5458,"mutex_wait_us":5,"thread_start_us":5334,"threads_started":1}
I20260928 15:47:35.770071 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.754143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.13.5.67:36713)
0928 15:47:35.760441 (+ 6298us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.760451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.760483 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:47:35.760581 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.760589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.760849 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.761066 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:35.761912 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.763393 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.766953 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.768916 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.769117 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.769369 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.769531 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.769692 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6109,"thread_start_us":5953,"threads_started":1}
I20260928 15:47:35.769809 27358 tablet_service.cc:1467] Processing CreateTablet for tablet e40346ad60a3417497f3cbdc6f646a86 (DEFAULT_TABLE table=test-table [id=bf975ca5ba7e4391a86b76c6154b20a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:35.769809 27494 tablet_service.cc:1467] Processing CreateTablet for tablet e40346ad60a3417497f3cbdc6f646a86 (DEFAULT_TABLE table=test-table [id=bf975ca5ba7e4391a86b76c6154b20a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:35.771451 27358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40346ad60a3417497f3cbdc6f646a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:35.771467 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40346ad60a3417497f3cbdc6f646a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:35.773764 27629 tablet_service.cc:1467] Processing CreateTablet for tablet e40346ad60a3417497f3cbdc6f646a86 (DEFAULT_TABLE table=test-table [id=bf975ca5ba7e4391a86b76c6154b20a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:35.774996 27629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40346ad60a3417497f3cbdc6f646a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:35.786777 27713 tablet_bootstrap.cc:492] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Bootstrap starting.
I20260928 15:47:35.788887 27715 tablet_bootstrap.cc:492] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Bootstrap starting.
I20260928 15:47:35.789695 27714 tablet_bootstrap.cc:492] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Bootstrap starting.
I20260928 15:47:35.791632 27713 tablet_bootstrap.cc:654] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:35.792585 27715 tablet_bootstrap.cc:654] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:35.793020 27713 log.cc:824] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:35.793613 27714 tablet_bootstrap.cc:654] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:35.794025 27715 log.cc:824] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:35.794930 27714 log.cc:824] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:35.796512 27713 tablet_bootstrap.cc:492] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: No bootstrap required, opened a new log
I20260928 15:47:35.796799 27713 ts_tablet_manager.cc:1397] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 15:47:35.797187 27715 tablet_bootstrap.cc:492] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: No bootstrap required, opened a new log
I20260928 15:47:35.797451 27715 ts_tablet_manager.cc:1397] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:47:35.797703 27714 tablet_bootstrap.cc:492] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: No bootstrap required, opened a new log
I20260928 15:47:35.797945 27714 ts_tablet_manager.cc:1397] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 15:47:35.805810 27714 raft_consensus.cc:348] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.806209 27714 raft_consensus.cc:374] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:35.806336 27714 raft_consensus.cc:729] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010c6024902d491a9081b4ffd8998727, State: Initialized, Role: FOLLOWER
I20260928 15:47:35.806991 27714 consensus_queue.cc:260] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.808225 27559 heartbeater.cc:503] Master 127.13.5.126:38275 was elected leader, sending a full tablet report...
I20260928 15:47:35.808173 27715 raft_consensus.cc:348] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.808609 27715 raft_consensus.cc:374] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:35.808704 27715 raft_consensus.cc:729] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e7954c1adc4d79930d0ca433bb9853, State: Initialized, Role: FOLLOWER
I20260928 15:47:35.809079 27714 ts_tablet_manager.cc:1428] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 15:47:35.809372 27715 consensus_queue.cc:260] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.809639 27713 raft_consensus.cc:348] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.810092 27713 raft_consensus.cc:374] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:35.810204 27713 raft_consensus.cc:729] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4107993bff64509b58337ead5767860, State: Initialized, Role: FOLLOWER
I20260928 15:47:35.810747 27713 consensus_queue.cc:260] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.811830 27694 heartbeater.cc:503] Master 127.13.5.126:38275 was elected leader, sending a full tablet report...
I20260928 15:47:35.812608 27713 ts_tablet_manager.cc:1428] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260928 15:47:35.814175 27715 ts_tablet_manager.cc:1428] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260928 15:47:35.826385 27425 tablet.cc:2369] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:47:35.880479 27721 raft_consensus.cc:482] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:47:35.880815 27721 raft_consensus.cc:504] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.882663 27721 leader_election.cc:288] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 010c6024902d491a9081b4ffd8998727 (127.13.5.66:43651), c1e7954c1adc4d79930d0ca433bb9853 (127.13.5.65:43301)
W20260928 15:47:35.883374 27560 tablet.cc:2369] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:47:35.890233 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.884035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:32873 (local address 127.13.5.66:43651)
0928 15:47:35.884627 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.884633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.884788 (+ 155us) server_negotiation.cc:407] Connection header received
0928 15:47:35.885073 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.885078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.885153 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.885252 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:35.885925 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.886650 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.888478 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.889232 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.889393 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.889489 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.889835 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.889970 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
I20260928 15:47:35.890255 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.883294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:55021 (local address 127.13.5.65:43301)
0928 15:47:35.883638 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.883644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.883669 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:47:35.884095 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.884100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.884190 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.884309 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:35.885182 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.886341 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.888348 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.889214 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.889379 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.889489 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.889845 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.889991 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260928 15:47:35.890300 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.883099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.13.5.67:55021)
0928 15:47:35.883455 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:35.883479 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:35.883650 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:35.884687 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:35.884695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:35.884734 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:35.884998 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.885011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.886506 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.886514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.888164 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.888177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.889427 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.889434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.889631 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.889696 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 15:47:35.889814 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:47:35.889957 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":56,"spinlock_wait_cycles":29440}
I20260928 15:47:35.890375 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.883203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43651 (local address 127.13.5.67:32873)
0928 15:47:35.884679 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:35.884697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:35.884838 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:35.885312 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:35.885318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:35.885350 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:35.885727 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.885742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.886806 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.886811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.888327 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:35.888341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.889443 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.889450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:35.889633 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.889692 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 15:47:35.889802 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:47:35.889934 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"mutex_wait_us":40,"thread_start_us":1108,"threads_started":1}
I20260928 15:47:35.891476 27514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40346ad60a3417497f3cbdc6f646a86" candidate_uuid: "f4107993bff64509b58337ead5767860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010c6024902d491a9081b4ffd8998727" is_pre_election: true
I20260928 15:47:35.891572 27379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40346ad60a3417497f3cbdc6f646a86" candidate_uuid: "f4107993bff64509b58337ead5767860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e7954c1adc4d79930d0ca433bb9853" is_pre_election: true
I20260928 15:47:35.892170 27514 raft_consensus.cc:2454] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4107993bff64509b58337ead5767860 in term 0.
I20260928 15:47:35.892170 27379 raft_consensus.cc:2454] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4107993bff64509b58337ead5767860 in term 0.
I20260928 15:47:35.892989 27582 leader_election.cc:302] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 010c6024902d491a9081b4ffd8998727, c1e7954c1adc4d79930d0ca433bb9853, f4107993bff64509b58337ead5767860; no voters:
I20260928 15:47:35.893538 27721 raft_consensus.cc:2781] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:47:35.893712 27721 raft_consensus.cc:482] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:47:35.893812 27721 raft_consensus.cc:3037] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:35.896210 27721 raft_consensus.cc:504] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.896833 27721 leader_election.cc:288] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [CANDIDATE]: Term 1 election: Requested vote from peers 010c6024902d491a9081b4ffd8998727 (127.13.5.66:43651), c1e7954c1adc4d79930d0ca433bb9853 (127.13.5.65:43301)
I20260928 15:47:35.897293 27514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40346ad60a3417497f3cbdc6f646a86" candidate_uuid: "f4107993bff64509b58337ead5767860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010c6024902d491a9081b4ffd8998727"
I20260928 15:47:35.897502 27514 raft_consensus.cc:3037] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:35.897428 27379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40346ad60a3417497f3cbdc6f646a86" candidate_uuid: "f4107993bff64509b58337ead5767860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e7954c1adc4d79930d0ca433bb9853"
I20260928 15:47:35.897714 27379 raft_consensus.cc:3037] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:35.899715 27514 raft_consensus.cc:2454] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4107993bff64509b58337ead5767860 in term 1.
I20260928 15:47:35.899971 27379 raft_consensus.cc:2454] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4107993bff64509b58337ead5767860 in term 1.
I20260928 15:47:35.900218 27582 leader_election.cc:302] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010c6024902d491a9081b4ffd8998727, f4107993bff64509b58337ead5767860; no voters:
I20260928 15:47:35.900501 27721 raft_consensus.cc:2781] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:47:35.900974 27721 raft_consensus.cc:686] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 1 LEADER]: Becoming Leader. State: Replica: f4107993bff64509b58337ead5767860, State: Running, Role: LEADER
I20260928 15:47:35.901482 27721 consensus_queue.cc:237] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } }
I20260928 15:47:35.906464 27232 catalog_manager.cc:5495] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4107993bff64509b58337ead5767860 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "f4107993bff64509b58337ead5767860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } health_report { overall_health: UNKNOWN } } }
W20260928 15:47:35.920728 27695 tablet.cc:2369] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:47:35.928937 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.922997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.13.5.67:36713)
0928 15:47:35.923176 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.923180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.923351 (+ 171us) server_negotiation.cc:407] Connection header received
0928 15:47:35.923537 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.923541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.923628 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.923751 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:47:35.924283 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.925083 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.926528 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.926928 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:35.927035 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:35.927698 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:47:35.928368 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:47:35.928484 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:35.928650 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 15:47:35.928725 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:47:35.937623 27514 raft_consensus.cc:1261] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 1 FOLLOWER]: Refusing update from remote peer f4107993bff64509b58337ead5767860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:47:35.937625 27379 raft_consensus.cc:1261] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 1 FOLLOWER]: Refusing update from remote peer f4107993bff64509b58337ead5767860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:47:35.938680 27723 consensus_queue.cc:1035] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:47:35.939086 27721 consensus_queue.cc:1035] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:47:35.954202 27727 mvcc.cc:205] Tried to move back new op lower bound from 7334340427510161408 to 7334340427376861184. Current Snapshot: MvccSnapshot[applied={T|T < 7334340427510161408}]
I20260928 15:47:35.975646 27232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:47:35.975970 27232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:47:35.980055 27232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Sending DeleteTablet for 3 replicas of tablet e40346ad60a3417497f3cbdc6f646a86
I20260928 15:47:35.981395 27629 tablet_service.cc:1514] Processing DeleteTablet for tablet e40346ad60a3417497f3cbdc6f646a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:35 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:55718
I20260928 15:47:35.981439 27494 tablet_service.cc:1514] Processing DeleteTablet for tablet e40346ad60a3417497f3cbdc6f646a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:35 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:34672
I20260928 15:47:35.981698 27358 tablet_service.cc:1514] Processing DeleteTablet for tablet e40346ad60a3417497f3cbdc6f646a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:35 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:45816
I20260928 15:47:35.982447 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 27295
I20260928 15:47:36.001099 27739 tablet_replica.cc:330] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: stopping tablet replica
I20260928 15:47:36.001767 27739 raft_consensus.cc:2229] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:47:36.002411 27739 raft_consensus.cc:2258] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:36.003327 27737 tablet_replica.cc:330] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: stopping tablet replica
I20260928 15:47:36.003557 27738 tablet_replica.cc:330] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: stopping tablet replica
I20260928 15:47:36.003911 27737 raft_consensus.cc:2229] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:47:36.004002 27738 raft_consensus.cc:2229] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:47:36.004539 27738 raft_consensus.cc:2258] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:36.004710 27737 raft_consensus.cc:2258] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:36.006996 27738 ts_tablet_manager.cc:1905] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:36.007362 27737 ts_tablet_manager.cc:1905] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:36.009572 27739 ts_tablet_manager.cc:1905] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:47:36 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:47:36.015322 27738 ts_tablet_manager.cc:1918] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:47:36.015597 27738 log.cc:1196] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/e40346ad60a3417497f3cbdc6f646a86
I20260928 15:47:36.016618 27738 ts_tablet_manager.cc:1939] T e40346ad60a3417497f3cbdc6f646a86 P 010c6024902d491a9081b4ffd8998727: Deleting consensus metadata
I20260928 15:47:36.018554 27737 ts_tablet_manager.cc:1918] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:47:36.018725 27737 log.cc:1196] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/e40346ad60a3417497f3cbdc6f646a86
I20260928 15:47:36.018625 27217 catalog_manager.cc:4841] TS 010c6024902d491a9081b4ffd8998727 (127.13.5.66:43651): tablet e40346ad60a3417497f3cbdc6f646a86 (table test-table [id=bf975ca5ba7e4391a86b76c6154b20a9]) successfully deleted
I20260928 15:47:36.019364 27737 ts_tablet_manager.cc:1939] T e40346ad60a3417497f3cbdc6f646a86 P f4107993bff64509b58337ead5767860: Deleting consensus metadata
I20260928 15:47:36.019645 27739 ts_tablet_manager.cc:1918] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:47:36.019912 27739 log.cc:1196] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/e40346ad60a3417497f3cbdc6f646a86
I20260928 15:47:36.020524 27739 ts_tablet_manager.cc:1939] T e40346ad60a3417497f3cbdc6f646a86 P c1e7954c1adc4d79930d0ca433bb9853: Deleting consensus metadata
I20260928 15:47:36.021210 27217 catalog_manager.cc:4841] TS f4107993bff64509b58337ead5767860 (127.13.5.67:36713): tablet e40346ad60a3417497f3cbdc6f646a86 (table test-table [id=bf975ca5ba7e4391a86b76c6154b20a9]) successfully deleted
I20260928 15:47:36.021818 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:35.984107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.13.5.65:43301)
0928 15:47:35.984364 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:35.984371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:35.992757 (+ 8386us) server_negotiation.cc:407] Connection header received
0928 15:47:35.993465 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:35.993475 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:35.993626 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:35.993730 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:35.996642 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:35.998250 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:36.005012 (+ 6762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:36.005672 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:36.005800 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:36.012042 (+ 6242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:36.012058 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:36.012076 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:36.012138 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:36.016717 (+ 4579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:36.017758 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:36.017767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:36.017770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:36.017958 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:36.018899 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:36.018908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:36.018912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:36.019510 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:36.019835 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:36.021251 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0928 15:47:36.021524 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:47:36.022541 27419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:36.023015 27219 catalog_manager.cc:4841] TS c1e7954c1adc4d79930d0ca433bb9853 (127.13.5.65:43301): tablet e40346ad60a3417497f3cbdc6f646a86 (table test-table [id=bf975ca5ba7e4391a86b76c6154b20a9]) successfully deleted
I20260928 15:47:36.116775 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 27295
I20260928 15:47:36.141112 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 27430
Sep 28 15:47:36 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:47:36.158010 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:36.142202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.13.5.66:43651)
0928 15:47:36.142397 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:36.142402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:36.142417 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:47:36.142470 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:36.142474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:36.142540 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:36.142666 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:36.144088 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:36.144837 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:36.146263 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:36.146687 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:36.146844 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:36.151278 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:36.151292 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:36.151307 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:36.151344 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:36.154645 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:36.155489 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:36.155497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:36.155499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:36.155629 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:36.156116 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:36.156122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:36.156125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:36.156712 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:36.157015 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:36.157352 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 15:47:36.157590 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:47:36.158681 27554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:36.251843 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 27430
I20260928 15:47:36.276995 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 27565
Sep 28 15:47:36 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:47:36.295446 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:36.277799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.13.5.67:36713)
0928 15:47:36.277987 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:36.277991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:36.278058 (+ 67us) server_negotiation.cc:407] Connection header received
0928 15:47:36.278259 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:36.278263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:36.278330 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:36.278414 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:36.280206 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:36.280993 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:36.282884 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:36.283326 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:36.283516 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:36.288349 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:36.288365 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:36.288382 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:36.288423 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:36.291860 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:36.292758 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:36.292773 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:36.292775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:36.292969 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:36.293477 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:36.293484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:36.293489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:36.294023 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:36.294424 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:36.294709 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 15:47:36.294971 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:47:36.296231 27689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:36.421713 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 27565
I20260928 15:47:36.442905 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 27200
I20260928 15:47:36.444219 27262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:36.557065 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 27200
I20260928 15:47:36.580276 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:38275
--webserver_interface=127.13.5.126
--webserver_port=35313
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:36.934767 27745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:36.935081 27745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:36.935263 27745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:36.942708 27745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:36.942823 27745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:36.942914 27745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:36.942998 27745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:36.943066 27745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:36.953069 27745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:38275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=35313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:36.954440 27745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:36.955778 27745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:36.980060 27754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:36.979980 27752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:36.980259 27751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:36.980968 27745 server_base.cc:1034] running on GCE node
I20260928 15:47:36.981945 27745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:36.983783 27745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:36.985071 27745 hybrid_clock.cc:648] HybridClock initialized: now 1790610456985014 us; error 42 us; skew 500 ppm
I20260928 15:47:36.985627 27745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:36 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610456, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:36.989620 27745 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:47:36.991412 27745 webserver.cc:460] Webserver started at http://127.13.5.126:35313/ using document root <none> and password file <none>
I20260928 15:47:36.992290 27745 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:36.992479 27745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:36.998847 27745 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 15:47:37.002089 27761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:37.003665 27745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:47:37.003963 27745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "af039d6ff1864cf793c47ee56f4a2de0"
format_stamp: "Formatted at 2026-09-28 15:47:33 on dist-test-slave-j968"
server_key: "2cc946721e08c351a4e12d7b196614c2"
server_key_iv: "ff4d7323cb298201e3c67ac3279ddb8e"
server_key_version: "encryptionkey@0"
I20260928 15:47:37.005163 27745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:37.027572 27745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:37.032419 27745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:37.032743 27745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:37.053759 27745 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:38275
I20260928 15:47:37.053783 27812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:38275 every 8 connection(s)
I20260928 15:47:37.055593 27745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:47:37.063169 27813 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:47:37.065532 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 27745
I20260928 15:47:37.066347 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:43301
--local_ip_for_outbound_sockets=127.13.5.65
--tserver_master_addrs=127.13.5.126:38275
--webserver_port=35959
--webserver_interface=127.13.5.65
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:47:37.066771 27813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Bootstrap starting.
I20260928 15:47:37.087353 27813 log.cc:824] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:37.100184 27813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:47:37.100754 27813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Bootstrap complete.
I20260928 15:47:37.111033 27813 raft_consensus.cc:348] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:37.111940 27813 raft_consensus.cc:729] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af039d6ff1864cf793c47ee56f4a2de0, State: Initialized, Role: FOLLOWER
I20260928 15:47:37.112587 27813 consensus_queue.cc:260] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:37.112846 27813 raft_consensus.cc:388] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:47:37.112983 27813 raft_consensus.cc:482] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:47:37.113171 27813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:47:37.117190 27813 raft_consensus.cc:504] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:37.117755 27813 leader_election.cc:302] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af039d6ff1864cf793c47ee56f4a2de0; no voters:
I20260928 15:47:37.118458 27813 leader_election.cc:288] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:47:37.118614 27817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:47:37.119884 27817 raft_consensus.cc:686] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 2 LEADER]: Becoming Leader. State: Replica: af039d6ff1864cf793c47ee56f4a2de0, State: Running, Role: LEADER
I20260928 15:47:37.120474 27817 consensus_queue.cc:237] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:37.121029 27813 sys_catalog.cc:564] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:37.123042 27819 sys_catalog.cc:455] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af039d6ff1864cf793c47ee56f4a2de0. Latest consensus state: current_term: 2 leader_uuid: "af039d6ff1864cf793c47ee56f4a2de0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } } }
I20260928 15:47:37.123406 27819 sys_catalog.cc:458] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:37.132091 27818 sys_catalog.cc:455] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "af039d6ff1864cf793c47ee56f4a2de0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } } }
I20260928 15:47:37.132717 27818 sys_catalog.cc:458] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:37.133520 27822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:47:37.144099 27822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:47:37.146008 27822 catalog_manager.cc:1259] Loaded cluster ID: 7a991cbddfe54bd7a444607435c87f94
I20260928 15:47:37.146143 27822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:47:37.150389 27822 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 15:47:37.176908 27822 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 15:47:37.191815 27822 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:47:37.194271 27822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Loaded TSK: 0
I20260928 15:47:37.196275 27822 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:47:37.453320 27815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:37.453626 27815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:37.453780 27815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:37.461499 27815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:37.461616 27815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:37.461793 27815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:47:37.472168 27815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:43301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=35959
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:37.473546 27815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:37.474949 27815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:37.488637 27841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:37.488693 27840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:37.489193 27843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:37.489683 27815 server_base.cc:1034] running on GCE node
I20260928 15:47:37.490525 27815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:37.491822 27815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:37.493101 27815 hybrid_clock.cc:648] HybridClock initialized: now 1790610457493017 us; error 68 us; skew 500 ppm
I20260928 15:47:37.493503 27815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:37 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610457, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:37.497607 27815 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:47:37.499266 27815 webserver.cc:460] Webserver started at http://127.13.5.65:35959/ using document root <none> and password file <none>
I20260928 15:47:37.500135 27815 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:37.500303 27815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:37.506381 27815 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260928 15:47:37.509814 27850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:37.511636 27815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:47:37.511827 27815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "c1e7954c1adc4d79930d0ca433bb9853"
format_stamp: "Formatted at 2026-09-28 15:47:34 on dist-test-slave-j968"
server_key: "b48b8e01f08d5d8ba4e290932deab758"
server_key_iv: "01502c140bfda56f1bcc1d9be4ce0d68"
server_key_version: "encryptionkey@0"
I20260928 15:47:37.512938 27815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:37.530699 27815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:37.535319 27815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:37.535629 27815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:37.536899 27815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:37.538417 27815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:37.538511 27815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:37.538666 27815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:37.538767 27815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:37.574998 27815 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43301
I20260928 15:47:37.575143 27963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43301 every 8 connection(s)
I20260928 15:47:37.576705 27815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:47:37.582921 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 27815
I20260928 15:47:37.583730 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:43651
--local_ip_for_outbound_sockets=127.13.5.66
--tserver_master_addrs=127.13.5.126:38275
--webserver_port=46423
--webserver_interface=127.13.5.66
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:47:37 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610457, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:37.604795 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:37.580230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:41753)
0928 15:47:37.581082 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:37.581135 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:37.583198 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:37.585118 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:37.585127 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:37.585697 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:37.586287 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:37.586305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:37.588930 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:37.588960 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:37.590271 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:37.590285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:37.591849 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:37.591859 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:37.592246 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:37.593204 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:37.593241 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:37.596189 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:37.599916 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:37.599926 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:37.599943 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:37.600415 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:37.600905 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:37.600910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:37.600914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:37.601135 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:37.602045 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:37.602053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:37.602432 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 15:47:37.602708 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 15:47:37.603325 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":364,"threads_started":1}
I20260928 15:47:37.604795 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:37.579935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:41753 (local address 127.13.5.126:38275)
0928 15:47:37.580604 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:37.580616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:37.581268 (+ 652us) server_negotiation.cc:407] Connection header received
0928 15:47:37.583957 (+ 2689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:37.583981 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:37.584492 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:37.584939 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:37.586569 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:37.588732 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:37.590437 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:37.591474 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:37.591627 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:37.596525 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:37.596535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:37.596553 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:37.596604 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:37.599701 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:37.600592 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:37.600600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:37.600606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:37.600775 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:37.601263 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:37.601273 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:37.601275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:37.601893 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:37.602116 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:37.602947 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 15:47:37.603360 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":313,"threads_started":1}
I20260928 15:47:37.608062 27964 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38275
I20260928 15:47:37.608413 27964 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:37.609421 27964 heartbeater.cc:511] Master 127.13.5.126:38275 requested a full tablet report, sending...
I20260928 15:47:37.611820 27778 ts_manager.cc:194] Registered new tserver with Master: c1e7954c1adc4d79930d0ca433bb9853 (127.13.5.65:43301)
I20260928 15:47:37.613366 27778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:41753
W20260928 15:47:37.943022 27968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:37.943401 27968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:37.943572 27968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:37.951074 27968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:37.951249 27968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:37.951440 27968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:47:37.961905 27968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=46423
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:37.963366 27968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:37.964764 27968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:37.978626 27973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:37.978582 27976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:37.978659 27974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:37.979578 27968 server_base.cc:1034] running on GCE node
I20260928 15:47:37.980304 27968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:37.981298 27968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:37.982568 27968 hybrid_clock.cc:648] HybridClock initialized: now 1790610457982501 us; error 59 us; skew 500 ppm
I20260928 15:47:37.982910 27968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:37 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610457, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:37.986282 27968 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:47:37.987962 27968 webserver.cc:460] Webserver started at http://127.13.5.66:46423/ using document root <none> and password file <none>
I20260928 15:47:37.988674 27968 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:37.988837 27968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:37.994438 27968 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:47:37.997622 27983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:37.999064 27968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:47:37.999264 27968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "010c6024902d491a9081b4ffd8998727"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "b782ab04666a10c0d3eed00c7491b83c"
server_key_iv: "7abdd0fefc0debc89f7a135c3083e6d8"
server_key_version: "encryptionkey@0"
I20260928 15:47:38.000488 27968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:38.030187 27968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:38.035224 27968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:38.035553 27968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:38.036765 27968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:38.038504 27968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:38.038653 27968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:38.038803 27968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:38.038928 27968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:38.078832 27968 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43651
I20260928 15:47:38.078930 28095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43651 every 8 connection(s)
I20260928 15:47:38.080509 27968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:47:38.082046 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 27968
I20260928 15:47:38.082742 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:36713
--local_ip_for_outbound_sockets=127.13.5.67
--tserver_master_addrs=127.13.5.126:38275
--webserver_port=41369
--webserver_interface=127.13.5.67
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:47:38 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610457, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:38.110708 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.083486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:47355 (local address 127.13.5.126:38275)
0928 15:47:38.083737 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.083747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.088873 (+ 5126us) server_negotiation.cc:407] Connection header received
0928 15:47:38.091358 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.091363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.091484 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.091578 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:38.093991 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.094702 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.097659 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.098157 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.098263 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.103546 (+ 5283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:38.103554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:38.103559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:38.103601 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:38.106871 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:38.107762 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:38.107769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:38.107773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:38.108029 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:38.108524 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:38.108529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:38.108533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:38.108942 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:38.109135 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.110050 (+ 915us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.110338 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:47:38.111816 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.084068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:47355)
0928 15:47:38.088621 (+ 4553us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:38.088682 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:38.090995 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:38.091929 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:38.091946 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:38.092844 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:38.093762 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.093794 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.094887 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.094894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.097461 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.097484 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.098443 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.098453 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.098820 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.099752 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:38.099791 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:38.103240 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:38.107047 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:38.107058 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:38.107079 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:38.107596 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:38.108156 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:38.108162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:38.108166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:38.108383 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:38.109153 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:38.109166 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:38.109551 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 15:47:38.109890 (+ 339us) client_negotiation.cc:240] Negotiation successful
0928 15:47:38.110532 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4168,"thread_start_us":3942,"threads_started":1}
I20260928 15:47:38.113608 28096 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38275
I20260928 15:47:38.113955 28096 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:38.114755 28096 heartbeater.cc:511] Master 127.13.5.126:38275 requested a full tablet report, sending...
I20260928 15:47:38.116426 27778 ts_manager.cc:194] Registered new tserver with Master: 010c6024902d491a9081b4ffd8998727 (127.13.5.66:43651)
I20260928 15:47:38.117511 27778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:47355
W20260928 15:47:38.432099 28100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:38.432461 28100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:38.432631 28100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:38.439852 28100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:38.440038 28100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:38.440227 28100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:47:38.450166 28100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:36713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=41369
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:38.451583 28100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:38.453051 28100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:38.467298 28108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:38.467322 28105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:38.467386 28106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:38.468154 28100 server_base.cc:1034] running on GCE node
I20260928 15:47:38.468779 28100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:38.469798 28100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:38.471029 28100 hybrid_clock.cc:648] HybridClock initialized: now 1790610458470954 us; error 61 us; skew 500 ppm
I20260928 15:47:38.471382 28100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:38 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610458, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:38.474692 28100 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:47:38.476315 28100 webserver.cc:460] Webserver started at http://127.13.5.67:41369/ using document root <none> and password file <none>
I20260928 15:47:38.477032 28100 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:38.477193 28100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:38.483157 28100 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 15:47:38.486268 28115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:38.487811 28100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:47:38.488049 28100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "f4107993bff64509b58337ead5767860"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "ef6b0cea4496ab09574914e1d524434e"
server_key_iv: "09eae94ddb7be650d9baac3fb2950ca4"
server_key_version: "encryptionkey@0"
I20260928 15:47:38.489131 28100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:38.507058 28100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:38.512276 28100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:38.512751 28100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:38.513972 28100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:38.515374 28100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:38.515471 28100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:38.515578 28100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:38.515683 28100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:38.551128 28100 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:36713
I20260928 15:47:38.551213 28229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:36713 every 8 connection(s)
I20260928 15:47:38.552721 28100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:47:38.562081 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 28100
Sep 28 15:47:38 dist-test-slave-j968 krb5kdc[27187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610458, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:38.574290 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.555572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:41515 (local address 127.13.5.126:38275)
0928 15:47:38.555774 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.555779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.557122 (+ 1343us) server_negotiation.cc:407] Connection header received
0928 15:47:38.558914 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.558919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.558990 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.559123 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:38.560664 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.561425 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.563377 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.564105 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.564280 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.567908 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:38.567916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:38.567921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:38.567968 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:38.570747 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:38.571507 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:38.571513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:38.571517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:38.571663 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:38.572188 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:38.572197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:38.572200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:38.572603 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:38.572775 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.573658 (+ 883us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.573897 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:47:38.576035 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.556144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:41515)
0928 15:47:38.556934 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:38.557004 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:38.558655 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:38.559349 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:38.559357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:38.559859 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:38.560507 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.560524 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.561572 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.561576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.563172 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.563189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.564336 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.564343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.564694 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.565402 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:38.565425 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:38.567632 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:38.570898 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:38.570907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:38.570924 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:38.571353 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:38.571784 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:38.571791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:38.571794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:38.572069 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:38.572741 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:38.572749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:38.573124 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 15:47:38.573542 (+ 418us) client_negotiation.cc:240] Negotiation successful
0928 15:47:38.574117 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":283,"threads_started":1}
I20260928 15:47:38.577837 28230 heartbeater.cc:347] Connected to a master server at 127.13.5.126:38275
I20260928 15:47:38.578190 28230 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:38.579134 28230 heartbeater.cc:511] Master 127.13.5.126:38275 requested a full tablet report, sending...
I20260928 15:47:38.580782 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.563036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.13.5.126:38275)
0928 15:47:38.563615 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.563621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.563851 (+ 230us) server_negotiation.cc:407] Connection header received
0928 15:47:38.564095 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.564106 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.564169 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.564293 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:38.566022 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.566759 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.568315 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.568787 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.568935 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.571657 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:38.571662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:38.571665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:38.571696 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:38.574804 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:38.575967 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:38.575972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:38.575975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:38.576119 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:38.576807 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:38.576817 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:38.576821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:38.577225 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:38.577477 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.580142 (+ 2665us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.580404 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
I20260928 15:47:38.580945 27778 ts_manager.cc:194] Registered new tserver with Master: f4107993bff64509b58337ead5767860 (127.13.5.67:36713)
I20260928 15:47:38.581974 27778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:41515
I20260928 15:47:38.583495 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:47:38.590852 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.584622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.13.5.126:38275)
0928 15:47:38.584820 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.584826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.585360 (+ 534us) server_negotiation.cc:407] Connection header received
0928 15:47:38.585647 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.585651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.585774 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.585922 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:47:38.586382 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.587064 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.588695 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.589123 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.589271 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.589778 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:47:38.590268 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:47:38.590428 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.590506 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.590591 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:47:38.592092 27777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:47:38.600236 27777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:47:38.616918 27964 heartbeater.cc:503] Master 127.13.5.126:38275 was elected leader, sending a full tablet report...
I20260928 15:47:38.626412 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.614230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.13.5.67:36713)
0928 15:47:38.617822 (+ 3592us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.617831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.617864 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:47:38.617973 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.617981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.618233 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.619180 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:38.619441 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.621210 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.623934 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.625309 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.625452 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.625693 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.626030 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.626165 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3389,"thread_start_us":3483,"threads_started":1}
I20260928 15:47:38.626832 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.613533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:36713 (local address 127.0.0.1:33710)
0928 15:47:38.617120 (+ 3587us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:38.617139 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:38.617248 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:38.618470 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:38.618475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:38.618499 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:38.618981 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.618996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.621417 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.621426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.623732 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.623747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.625524 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.625530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.625759 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.625829 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 15:47:38.625982 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 15:47:38.626188 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3469,"mutex_wait_us":4,"thread_start_us":881,"threads_started":1}
I20260928 15:47:38.628047 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.613366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46660)
0928 15:47:38.614881 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:38.614942 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:38.615144 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:38.617932 (+ 2788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:38.617939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:38.617970 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:38.618485 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.618497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.620803 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.620808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.623576 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.623586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.627170 (+ 3584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.627179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.627370 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.627414 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 15:47:38.627507 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:47:38.627649 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":21,"thread_start_us":4211,"threads_started":1}
I20260928 15:47:38.629963 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.618276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43651 (local address 127.0.0.1:51548)
0928 15:47:38.621931 (+ 3655us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:38.621954 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:38.622132 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:38.622804 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:38.622809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:38.622834 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:38.623300 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.623311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.625124 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.625127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.627107 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.627120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.629176 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.629182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.629360 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.629395 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:47:38.629488 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:47:38.629617 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3507,"thread_start_us":3286,"threads_started":1}
I20260928 15:47:38.629981 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.618762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.13.5.66:43651)
0928 15:47:38.619346 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.619358 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.622077 (+ 2719us) server_negotiation.cc:407] Connection header received
0928 15:47:38.622319 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.622329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.622591 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.622793 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:38.623500 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.624948 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.627277 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.628951 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.629082 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.629285 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.629550 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.629705 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":270,"threads_started":1}
I20260928 15:47:38.631394 28164 tablet_service.cc:1467] Processing CreateTablet for tablet 31395f32b4eb45b39ae7f39c0b015286 (DEFAULT_TABLE table=test-table [id=3c888e61eaf74d8fbd3dba6ffbfa19ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:38.632280 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.613695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.13.5.65:43301)
0928 15:47:38.615819 (+ 2124us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.615832 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.615886 (+ 54us) server_negotiation.cc:407] Connection header received
0928 15:47:38.615988 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.615995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.617405 (+ 1410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.617634 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:38.618762 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.620646 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.623729 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.625309 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.625445 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.625693 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.627582 (+ 1889us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.627762 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1942,"thread_start_us":1817,"threads_started":1}
I20260928 15:47:38.633821 28164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31395f32b4eb45b39ae7f39c0b015286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:38.635707 27899 tablet_service.cc:1467] Processing CreateTablet for tablet 31395f32b4eb45b39ae7f39c0b015286 (DEFAULT_TABLE table=test-table [id=3c888e61eaf74d8fbd3dba6ffbfa19ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:38.635754 28031 tablet_service.cc:1467] Processing CreateTablet for tablet 31395f32b4eb45b39ae7f39c0b015286 (DEFAULT_TABLE table=test-table [id=3c888e61eaf74d8fbd3dba6ffbfa19ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:47:38.637094 27899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31395f32b4eb45b39ae7f39c0b015286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:38.637161 28031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31395f32b4eb45b39ae7f39c0b015286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:38.653748 28247 tablet_bootstrap.cc:492] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Bootstrap starting.
I20260928 15:47:38.654218 28246 tablet_bootstrap.cc:492] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Bootstrap starting.
I20260928 15:47:38.657698 28246 tablet_bootstrap.cc:654] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:38.658756 28246 log.cc:824] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:38.659725 28247 tablet_bootstrap.cc:654] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:38.659821 28248 tablet_bootstrap.cc:492] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Bootstrap starting.
I20260928 15:47:38.661134 28247 log.cc:824] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:38.662179 28246 tablet_bootstrap.cc:492] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: No bootstrap required, opened a new log
I20260928 15:47:38.662488 28246 ts_tablet_manager.cc:1397] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:47:38.663663 28248 tablet_bootstrap.cc:654] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:38.664839 28248 log.cc:824] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:38.665552 28247 tablet_bootstrap.cc:492] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: No bootstrap required, opened a new log
I20260928 15:47:38.665822 28247 ts_tablet_manager.cc:1397] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260928 15:47:38.667532 28248 tablet_bootstrap.cc:492] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: No bootstrap required, opened a new log
I20260928 15:47:38.667807 28248 ts_tablet_manager.cc:1397] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:47:38.673565 28246 raft_consensus.cc:348] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
I20260928 15:47:38.674073 28246 raft_consensus.cc:374] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:38.674175 28246 raft_consensus.cc:729] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e7954c1adc4d79930d0ca433bb9853, State: Initialized, Role: FOLLOWER
I20260928 15:47:38.674898 28246 consensus_queue.cc:260] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
I20260928 15:47:38.675956 28248 raft_consensus.cc:348] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
I20260928 15:47:38.676419 28248 raft_consensus.cc:374] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:38.676530 28248 raft_consensus.cc:729] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010c6024902d491a9081b4ffd8998727, State: Initialized, Role: FOLLOWER
I20260928 15:47:38.676469 28247 raft_consensus.cc:348] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
I20260928 15:47:38.676898 28247 raft_consensus.cc:374] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:38.676977 28246 ts_tablet_manager.cc:1428] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260928 15:47:38.676993 28247 raft_consensus.cc:729] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4107993bff64509b58337ead5767860, State: Initialized, Role: FOLLOWER
I20260928 15:47:38.677228 28248 consensus_queue.cc:260] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
I20260928 15:47:38.677652 28247 consensus_queue.cc:260] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
I20260928 15:47:38.678738 28096 heartbeater.cc:503] Master 127.13.5.126:38275 was elected leader, sending a full tablet report...
I20260928 15:47:38.679481 28248 ts_tablet_manager.cc:1428] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260928 15:47:38.680631 28230 heartbeater.cc:503] Master 127.13.5.126:38275 was elected leader, sending a full tablet report...
I20260928 15:47:38.681313 28247 ts_tablet_manager.cc:1428] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260928 15:47:38.804632 28254 raft_consensus.cc:482] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260928 15:47:38.804745 28231 tablet.cc:2369] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:47:38.804996 28254 raft_consensus.cc:504] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
I20260928 15:47:38.807358 28254 leader_election.cc:288] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e7954c1adc4d79930d0ca433bb9853 (127.13.5.65:43301), 010c6024902d491a9081b4ffd8998727 (127.13.5.66:43651)
I20260928 15:47:38.813318 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.807944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:46683 (local address 127.13.5.65:43301)
0928 15:47:38.808162 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.808168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.808190 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:47:38.808281 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.808287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.808393 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.808515 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:38.809046 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.809827 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.811477 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.812412 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.812572 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.812679 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.812938 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.813066 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:47:38.813330 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.807535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.13.5.67:46683)
0928 15:47:38.807786 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:38.807810 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:38.807988 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:38.808571 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:38.808577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:38.808613 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:38.808863 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.808877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.809970 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.809978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.811289 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.811301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.812586 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.812591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.812776 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.812802 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:47:38.812912 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:47:38.813055 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:47:38.814337 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.808022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:43651 (local address 127.13.5.67:56193)
0928 15:47:38.808791 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:38.808812 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:38.808954 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:38.809395 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:38.809408 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:38.809469 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:47:38.809754 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.809763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.810894 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.810899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.812456 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:38.812468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.813695 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.813703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:38.813876 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.813916 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:47:38.814016 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:47:38.814103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":412,"threads_started":1}
I20260928 15:47:38.814502 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.808115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:56193 (local address 127.13.5.66:43651)
0928 15:47:38.808357 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.808365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.808886 (+ 521us) server_negotiation.cc:407] Connection header received
0928 15:47:38.809125 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.809136 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.809258 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.809367 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:47:38.809910 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.810750 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.812603 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.813511 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.813640 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.813728 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.814081 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.814228 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:47:38.814626 27919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31395f32b4eb45b39ae7f39c0b015286" candidate_uuid: "f4107993bff64509b58337ead5767860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e7954c1adc4d79930d0ca433bb9853" is_pre_election: true
I20260928 15:47:38.815200 27919 raft_consensus.cc:2454] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4107993bff64509b58337ead5767860 in term 0.
I20260928 15:47:38.815424 28051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31395f32b4eb45b39ae7f39c0b015286" candidate_uuid: "f4107993bff64509b58337ead5767860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010c6024902d491a9081b4ffd8998727" is_pre_election: true
I20260928 15:47:38.815984 28051 raft_consensus.cc:2454] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4107993bff64509b58337ead5767860 in term 0.
I20260928 15:47:38.816035 28119 leader_election.cc:302] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e7954c1adc4d79930d0ca433bb9853, f4107993bff64509b58337ead5767860; no voters:
I20260928 15:47:38.816586 28254 raft_consensus.cc:2781] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:47:38.816776 28254 raft_consensus.cc:482] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:47:38.816891 28254 raft_consensus.cc:3037] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:38.820492 28254 raft_consensus.cc:504] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
I20260928 15:47:38.821359 28254 leader_election.cc:288] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [CANDIDATE]: Term 1 election: Requested vote from peers c1e7954c1adc4d79930d0ca433bb9853 (127.13.5.65:43301), 010c6024902d491a9081b4ffd8998727 (127.13.5.66:43651)
I20260928 15:47:38.821975 28051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31395f32b4eb45b39ae7f39c0b015286" candidate_uuid: "f4107993bff64509b58337ead5767860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010c6024902d491a9081b4ffd8998727"
I20260928 15:47:38.822227 28051 raft_consensus.cc:3037] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:38.823608 27919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31395f32b4eb45b39ae7f39c0b015286" candidate_uuid: "f4107993bff64509b58337ead5767860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e7954c1adc4d79930d0ca433bb9853"
I20260928 15:47:38.823827 27919 raft_consensus.cc:3037] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:38.825184 28051 raft_consensus.cc:2454] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4107993bff64509b58337ead5767860 in term 1.
I20260928 15:47:38.825644 28117 leader_election.cc:302] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010c6024902d491a9081b4ffd8998727, f4107993bff64509b58337ead5767860; no voters:
I20260928 15:47:38.825999 28254 raft_consensus.cc:2781] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:47:38.826447 28254 raft_consensus.cc:686] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 1 LEADER]: Becoming Leader. State: Replica: f4107993bff64509b58337ead5767860, State: Running, Role: LEADER
I20260928 15:47:38.826555 27919 raft_consensus.cc:2454] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4107993bff64509b58337ead5767860 in term 1.
I20260928 15:47:38.827035 28254 consensus_queue.cc:237] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } }
W20260928 15:47:38.829216 27965 tablet.cc:2369] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:47:38.832672 28097 tablet.cc:2369] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:47:38.832883 27777 catalog_manager.cc:5495] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4107993bff64509b58337ead5767860 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "f4107993bff64509b58337ead5767860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4107993bff64509b58337ead5767860" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 36713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 } health_report { overall_health: UNKNOWN } } }
I20260928 15:47:38.866017 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.860444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.13.5.67:36713)
0928 15:47:38.860685 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.860691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.860710 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:47:38.860972 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.860976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.861065 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.861226 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:47:38.861699 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.862447 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.863859 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.864312 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.864408 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.864962 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:47:38.865494 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:47:38.865613 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.865755 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.865821 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:47:38.874562 27919 raft_consensus.cc:1261] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 1 FOLLOWER]: Refusing update from remote peer f4107993bff64509b58337ead5767860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:47:38.874563 28051 raft_consensus.cc:1261] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 1 FOLLOWER]: Refusing update from remote peer f4107993bff64509b58337ead5767860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:47:38.875593 28254 consensus_queue.cc:1035] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e7954c1adc4d79930d0ca433bb9853" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:47:38.876012 28256 consensus_queue.cc:1035] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "010c6024902d491a9081b4ffd8998727" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:47:38.902822 27778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:47:38.903112 27778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:47:38.907321 27778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Sending DeleteTablet for 3 replicas of tablet 31395f32b4eb45b39ae7f39c0b015286
I20260928 15:47:38.908912 28164 tablet_service.cc:1514] Processing DeleteTablet for tablet 31395f32b4eb45b39ae7f39c0b015286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:38 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:33710
I20260928 15:47:38.908922 27899 tablet_service.cc:1514] Processing DeleteTablet for tablet 31395f32b4eb45b39ae7f39c0b015286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:38 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:46660
I20260928 15:47:38.909133 28031 tablet_service.cc:1514] Processing DeleteTablet for tablet 31395f32b4eb45b39ae7f39c0b015286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:47:38 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:51548
I20260928 15:47:38.909919 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 27815
I20260928 15:47:38.918406 28268 tablet_replica.cc:330] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: stopping tablet replica
I20260928 15:47:38.918890 28268 raft_consensus.cc:2229] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:47:38.919461 28268 raft_consensus.cc:2258] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:38.921346 28270 tablet_replica.cc:330] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: stopping tablet replica
I20260928 15:47:38.921837 28270 raft_consensus.cc:2229] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:47:38.922349 28270 raft_consensus.cc:2258] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:38.925530 28270 ts_tablet_manager.cc:1905] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:38.929200 28268 ts_tablet_manager.cc:1905] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:38.933588 28270 ts_tablet_manager.cc:1918] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:47:38.933804 28270 log.cc:1196] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/31395f32b4eb45b39ae7f39c0b015286
I20260928 15:47:38.934477 28270 ts_tablet_manager.cc:1939] T 31395f32b4eb45b39ae7f39c0b015286 P 010c6024902d491a9081b4ffd8998727: Deleting consensus metadata
I20260928 15:47:38.935855 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:38.911358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.13.5.65:43301)
0928 15:47:38.911557 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:38.911563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:38.911581 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:47:38.911640 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:38.911645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:38.911721 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:38.911819 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:38.913906 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.915153 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.917231 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:38.918004 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:38.918138 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:38.927604 (+ 9466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:38.927625 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:38.927643 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:38.927692 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:38.931823 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:38.932872 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:38.932883 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:38.932886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:38.933089 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:38.933571 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:38.933578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:38.933581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:38.934147 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:38.934486 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:38.934955 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 15:47:38.935213 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 15:47:38.936585 27959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:38.936621 27763 catalog_manager.cc:4841] TS 010c6024902d491a9081b4ffd8998727 (127.13.5.66:43651): tablet 31395f32b4eb45b39ae7f39c0b015286 (table test-table [id=3c888e61eaf74d8fbd3dba6ffbfa19ba]) successfully deleted
I20260928 15:47:38.939517 28269 tablet_replica.cc:330] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: stopping tablet replica
I20260928 15:47:38.940162 28269 raft_consensus.cc:2229] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:47:38.941058 28269 raft_consensus.cc:2258] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:47:38.944427 28269 ts_tablet_manager.cc:1905] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:38.952827 28269 ts_tablet_manager.cc:1918] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:47:38.953040 28269 log.cc:1196] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/31395f32b4eb45b39ae7f39c0b015286
I20260928 15:47:38.953698 28269 ts_tablet_manager.cc:1939] T 31395f32b4eb45b39ae7f39c0b015286 P f4107993bff64509b58337ead5767860: Deleting consensus metadata
I20260928 15:47:38.955821 27763 catalog_manager.cc:4841] TS f4107993bff64509b58337ead5767860 (127.13.5.67:36713): tablet 31395f32b4eb45b39ae7f39c0b015286 (table test-table [id=3c888e61eaf74d8fbd3dba6ffbfa19ba]) successfully deleted
I20260928 15:47:39.034008 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 27815
W20260928 15:47:39.059605 27765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.5.65:43301 (error 108)
I20260928 15:47:39.059820 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 27968
I20260928 15:47:39.066689 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.066209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46682)
0928 15:47:39.066366 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:39.066491 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:47:39.067046 27765 catalog_manager.cc:4584] TS c1e7954c1adc4d79930d0ca433bb9853 (127.13.5.65:43301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31395f32b4eb45b39ae7f39c0b015286: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
I20260928 15:47:39.080214 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.061484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.13.5.66:43651)
0928 15:47:39.061734 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:39.061743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:39.061770 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:47:39.061853 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:39.061862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:39.061969 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:39.062101 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:39.064722 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:39.065823 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:39.068228 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:39.068869 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:39.068995 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:39.072631 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:39.072651 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:39.072672 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:39.072728 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:39.076768 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:39.077730 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:39.077738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:39.077740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:39.077873 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:39.078423 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:39.078426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:39.078428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:39.078842 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:39.079167 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:39.079514 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 15:47:39.079764 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:47:39.081012 28091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:39.121145 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.120489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46686)
0928 15:47:39.120768 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:39.120895 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 15:47:39.122519 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.122025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46698)
0928 15:47:39.122215 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:39.122341 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 15:47:39.173929 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 27968
I20260928 15:47:39.183322 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.182713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46706)
0928 15:47:39.182983 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:39.183108 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 15:47:39.184587 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.184160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46722)
0928 15:47:39.184337 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:39.184436 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:47:39.194533 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 28100
I20260928 15:47:39.211670 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.195379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.13.5.67:36713)
0928 15:47:39.195555 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:39.195559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:39.195615 (+ 56us) server_negotiation.cc:407] Connection header received
0928 15:47:39.195849 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:39.195853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:39.195986 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:39.196086 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:39.198066 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:39.198886 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:39.200934 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:39.201430 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:39.201546 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:39.205023 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:39.205034 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:39.205049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:39.205090 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:39.208481 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:39.209455 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:39.209465 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:39.209468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:39.209670 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:39.210105 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:39.210112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:39.210114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:39.210507 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:39.210746 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:39.211110 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 15:47:39.211281 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:47:39.212461 28225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:39.265621 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.265030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46738)
0928 15:47:39.265260 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:39.265358 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 15:47:39.266769 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.266365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46742)
0928 15:47:39.266589 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:39.266657 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 15:47:39.336491 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 28100
I20260928 15:47:39.357122 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 27745
I20260928 15:47:39.358443 27808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:39.474931 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:39.474144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43301 (local address 127.0.0.1:46746)
0928 15:47:39.474432 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:39.474559 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 15:47:39.475227 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 27745
I20260928 15:47:39.497861 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:38275
--webserver_interface=127.13.5.126
--webserver_port=35313
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:38275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:39.842434 28277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:39.842819 28277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:39.842978 28277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:39.850313 28277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:39.850482 28277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:39.850597 28277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:39.850701 28277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:39.850780 28277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:39.861100 28277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:38275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:38275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=35313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:39.862562 28277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:39.863956 28277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:39.876291 28285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:39.876370 28282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:39.876370 28283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:39.877285 28277 server_base.cc:1034] running on GCE node
I20260928 15:47:39.878131 28277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:39.879576 28277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:39.880789 28277 hybrid_clock.cc:648] HybridClock initialized: now 1790610459880711 us; error 62 us; skew 500 ppm
I20260928 15:47:39.881117 28277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:39 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610459, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:39.884377 28277 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:47:39.885792 28277 webserver.cc:460] Webserver started at http://127.13.5.126:35313/ using document root <none> and password file <none>
I20260928 15:47:39.886447 28277 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:39.886546 28277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:39.892171 28277 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:47:39.895117 28292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:39.896703 28277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:47:39.896857 28277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "af039d6ff1864cf793c47ee56f4a2de0"
format_stamp: "Formatted at 2026-09-28 15:47:33 on dist-test-slave-j968"
server_key: "2cc946721e08c351a4e12d7b196614c2"
server_key_iv: "ff4d7323cb298201e3c67ac3279ddb8e"
server_key_version: "encryptionkey@0"
I20260928 15:47:39.897814 28277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:39.914315 28277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:39.918637 28277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:39.918874 28277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:39.939189 28277 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:38275
I20260928 15:47:39.939189 28343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:38275 every 8 connection(s)
I20260928 15:47:39.940984 28277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:47:39.942215 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 28277
I20260928 15:47:39.943054 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:43301
--local_ip_for_outbound_sockets=127.13.5.65
--tserver_master_addrs=127.13.5.126:38275
--webserver_port=35959
--webserver_interface=127.13.5.65
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:47:39.949021 28344 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:47:39.953475 28344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Bootstrap starting.
I20260928 15:47:39.978499 28344 log.cc:824] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:39.995756 28344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:47:39.996374 28344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0: Bootstrap complete.
I20260928 15:47:40.005998 28344 raft_consensus.cc:348] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:40.006901 28344 raft_consensus.cc:729] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af039d6ff1864cf793c47ee56f4a2de0, State: Initialized, Role: FOLLOWER
I20260928 15:47:40.007700 28344 consensus_queue.cc:260] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:40.008055 28344 raft_consensus.cc:388] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:47:40.008181 28344 raft_consensus.cc:482] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:47:40.008384 28344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:47:40.012688 28344 raft_consensus.cc:504] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:40.013303 28344 leader_election.cc:302] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af039d6ff1864cf793c47ee56f4a2de0; no voters:
I20260928 15:47:40.014137 28344 leader_election.cc:288] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:47:40.014292 28348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:47:40.015702 28348 raft_consensus.cc:686] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [term 3 LEADER]: Becoming Leader. State: Replica: af039d6ff1864cf793c47ee56f4a2de0, State: Running, Role: LEADER
I20260928 15:47:40.016376 28348 consensus_queue.cc:237] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } }
I20260928 15:47:40.016649 28344 sys_catalog.cc:564] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:40.019560 28349 sys_catalog.cc:455] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "af039d6ff1864cf793c47ee56f4a2de0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } } }
I20260928 15:47:40.020051 28349 sys_catalog.cc:458] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:40.020597 28350 sys_catalog.cc:455] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af039d6ff1864cf793c47ee56f4a2de0. Latest consensus state: current_term: 3 leader_uuid: "af039d6ff1864cf793c47ee56f4a2de0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af039d6ff1864cf793c47ee56f4a2de0" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 38275 } } }
I20260928 15:47:40.020953 28350 sys_catalog.cc:458] T 00000000000000000000000000000000 P af039d6ff1864cf793c47ee56f4a2de0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:40.029731 28356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:47:40.039290 28356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:47:40.040900 28356 catalog_manager.cc:1259] Loaded cluster ID: 7a991cbddfe54bd7a444607435c87f94
I20260928 15:47:40.041066 28356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 15:47:40.043265 28356 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe753ec19ad google::LogMessage::Fail() at ??:0
@ 0x7fe753ec488a google::LogMessage::SendToLog() at ??:0
@ 0x7fe753ec14de google::LogMessage::Flush() at ??:0
@ 0x7fe753ec2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe7698f4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe7698f1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe769979610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe769979439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe756b0d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe756b1091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe756b10799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe756aebeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe759dc56db start_thread at ??:0
@ 0x7fe752d5b61f clone at ??:0
W20260928 15:47:40.334494 28346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:40.334801 28346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:40.334965 28346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:40.343247 28346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:40.343380 28346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:40.343575 28346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:47:40.354003 28346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:43301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=35959
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:40.355397 28346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:40.356887 28346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:40.370973 28385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:40.370950 28388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:40.370913 28386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:40.372360 28346 server_base.cc:1034] running on GCE node
I20260928 15:47:40.373178 28346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:40.374276 28346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:40.375578 28346 hybrid_clock.cc:648] HybridClock initialized: now 1790610460375502 us; error 61 us; skew 500 ppm
I20260928 15:47:40.376071 28346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:40 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610460, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:40.379664 28346 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:47:40.381510 28346 webserver.cc:460] Webserver started at http://127.13.5.65:35959/ using document root <none> and password file <none>
I20260928 15:47:40.382308 28346 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:40.382470 28346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:40.389019 28346 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:47:40.392452 28395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:40.394174 28346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:47:40.394397 28346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "c1e7954c1adc4d79930d0ca433bb9853"
format_stamp: "Formatted at 2026-09-28 15:47:34 on dist-test-slave-j968"
server_key: "b48b8e01f08d5d8ba4e290932deab758"
server_key_iv: "01502c140bfda56f1bcc1d9be4ce0d68"
server_key_version: "encryptionkey@0"
I20260928 15:47:40.395510 28346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:40.409015 28346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:40.413750 28346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:40.414052 28346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:40.415064 28346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:40.417001 28402 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 15:47:40.422715 28402 ts_tablet_manager.cc:1852] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 15:47:40.423718 28402 ts_tablet_manager.cc:1905] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:47:40.431378 28402 ts_tablet_manager.cc:1918] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:47:40.431607 28402 log.cc:1196] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/31395f32b4eb45b39ae7f39c0b015286
I20260928 15:47:40.432144 28402 ts_tablet_manager.cc:1939] T 31395f32b4eb45b39ae7f39c0b015286 P c1e7954c1adc4d79930d0ca433bb9853: Deleting consensus metadata
I20260928 15:47:40.432615 28346 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 15:47:40.432716 28346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.001s
I20260928 15:47:40.432808 28346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:40.432857 28346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:40.468149 28346 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43301
I20260928 15:47:40.468242 28509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43301 every 8 connection(s)
I20260928 15:47:40.469581 28346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:47:40.473940 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 28346
I20260928 15:47:40.474608 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:43651
--local_ip_for_outbound_sockets=127.13.5.66
--tserver_master_addrs=127.13.5.126:38275
--webserver_port=46423
--webserver_interface=127.13.5.66
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:47:40.474917 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:40.472852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:45783)
0928 15:47:40.473640 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:40.473835 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":312,"threads_started":1}
W20260928 15:47:40.475706 28510 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:38275 (0 consecutive failures): Network error: Failed to ping master at 127.13.5.126:38275: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
I20260928 15:47:40.476747 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:40.476301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:46277)
0928 15:47:40.476523 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:40.476632 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:47:40.477602 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:40.477253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:45339)
0928 15:47:40.477393 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:40.477486 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:47:40.477911 28510 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:47:40.816601 28513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:40.816895 28513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:40.817088 28513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:40.824653 28513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:40.824777 28513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:40.824998 28513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:47:40.835090 28513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=46423
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:40.836539 28513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:40.838018 28513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:40.852774 28518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:40.852797 28519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:40.852761 28521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:40.853889 28513 server_base.cc:1034] running on GCE node
I20260928 15:47:40.854578 28513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:40.855641 28513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:40.856887 28513 hybrid_clock.cc:648] HybridClock initialized: now 1790610460856803 us; error 75 us; skew 500 ppm
I20260928 15:47:40.857246 28513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:40 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610460, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:40.860644 28513 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:47:40.862270 28513 webserver.cc:460] Webserver started at http://127.13.5.66:46423/ using document root <none> and password file <none>
I20260928 15:47:40.863008 28513 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:40.863161 28513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:40.869021 28513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:47:40.872390 28528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:40.873812 28513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:47:40.874010 28513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "010c6024902d491a9081b4ffd8998727"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "b782ab04666a10c0d3eed00c7491b83c"
server_key_iv: "7abdd0fefc0debc89f7a135c3083e6d8"
server_key_version: "encryptionkey@0"
I20260928 15:47:40.875059 28513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:40.893368 28513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:40.897699 28513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:40.897994 28513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:40.899160 28513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:40.900776 28513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:40.900905 28513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:40.901027 28513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:40.901119 28513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:40.935945 28513 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:43651
I20260928 15:47:40.936035 28641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:43651 every 8 connection(s)
I20260928 15:47:40.937517 28513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:47:40.939356 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 28513
I20260928 15:47:40.940305 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:36713
--local_ip_for_outbound_sockets=127.13.5.67
--tserver_master_addrs=127.13.5.126:38275
--webserver_port=41369
--webserver_interface=127.13.5.67
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:47:40.943167 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:40.941048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:48227)
0928 15:47:40.941944 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:40.942191 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":400,"threads_started":1}
W20260928 15:47:40.944082 28642 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:38275 (0 consecutive failures): Network error: Failed to ping master at 127.13.5.126:38275: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
I20260928 15:47:40.945029 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:40.944674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:51061)
0928 15:47:40.944842 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:40.944920 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 15:47:40.946069 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:40.945597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:38601)
0928 15:47:40.945787 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:40.945896 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 15:47:40.946348 28642 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:47:41.276944 28645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:41.277242 28645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:41.277338 28645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:41.284994 28645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:41.285120 28645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:41.285235 28645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:47:41.294872 28645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:36713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=41369
--tserver_master_addrs=127.13.5.126:38275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:41.296159 28645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:41.297465 28645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:41.311517 28650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:41.311431 28651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:41.311587 28653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:41.312544 28645 server_base.cc:1034] running on GCE node
I20260928 15:47:41.313248 28645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:41.314246 28645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:41.315490 28645 hybrid_clock.cc:648] HybridClock initialized: now 1790610461315415 us; error 61 us; skew 500 ppm
I20260928 15:47:41.315830 28645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:41 dist-test-slave-j968 krb5kdc[27187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610461, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:41.319172 28645 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:47:41.320796 28645 webserver.cc:460] Webserver started at http://127.13.5.67:41369/ using document root <none> and password file <none>
I20260928 15:47:41.321543 28645 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:41.321686 28645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:41.327502 28645 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 15:47:41.330799 28660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:41.332495 28645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:47:41.332705 28645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "f4107993bff64509b58337ead5767860"
format_stamp: "Formatted at 2026-09-28 15:47:35 on dist-test-slave-j968"
server_key: "ef6b0cea4496ab09574914e1d524434e"
server_key_iv: "09eae94ddb7be650d9baac3fb2950ca4"
server_key_version: "encryptionkey@0"
I20260928 15:47:41.333772 28645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:41.368322 28645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:41.372789 28645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:41.373098 28645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:41.374200 28645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:41.375602 28645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:41.375694 28645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:41.375859 28645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:41.376010 28645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 15:47:41.410136 28645 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:36713
I20260928 15:47:41.410220 28773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:36713 every 8 connection(s)
I20260928 15:47:41.411769 28645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:47:41.415124 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 28645
I20260928 15:47:41.417538 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:41.415079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:56917)
0928 15:47:41.416158 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:41.416471 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":458,"threads_started":1}
W20260928 15:47:41.418346 28774 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:38275 (0 consecutive failures): Network error: Failed to ping master at 127.13.5.126:38275: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
I20260928 15:47:41.419334 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:41.418958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:33397)
0928 15:47:41.419150 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:41.419220 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:47:41.420477 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:41.420078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:44901)
0928 15:47:41.420225 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:41.420301 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 15:47:41.420749 28774 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:47:41.479387 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:41.478536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:36567)
0928 15:47:41.479023 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:41.479137 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
I20260928 15:47:41.948145 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:41.947106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:50625)
0928 15:47:41.947668 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:41.947791 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":281,"threads_started":1}
I20260928 15:47:42.422207 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:42.421354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:54019)
0928 15:47:42.421865 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:42.421969 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
I20260928 15:47:42.481299 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:42.480531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:37211)
0928 15:47:42.481061 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:42.481140 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":310,"threads_started":1}
I20260928 15:47:42.950145 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:42.949140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:51589)
0928 15:47:42.949743 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:42.949865 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":271,"threads_started":1}
I20260928 15:47:43.423856 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:43.423121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:49119)
0928 15:47:43.423607 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:43.423689 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260928 15:47:43.483027 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:43.482261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:51953)
0928 15:47:43.482725 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:43.482818 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":228,"threads_started":1}
I20260928 15:47:43.951854 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:43.951123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:55775)
0928 15:47:43.951582 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:43.951666 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260928 15:47:44.425725 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:44.425010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:52777)
0928 15:47:44.425463 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:44.425553 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260928 15:47:44.484731 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:44.484008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:54247)
0928 15:47:44.484467 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:44.484560 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260928 15:47:44.953531 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:44.952785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:43251)
0928 15:47:44.953237 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:44.953317 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260928 15:47:45.427512 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:45.426736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:36971)
0928 15:47:45.427237 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:45.427321 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260928 15:47:45.486574 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:45.485779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:35873)
0928 15:47:45.486279 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:45.486381 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260928 15:47:45.955493 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:45.954696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:40441)
0928 15:47:45.955198 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:45.955283 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
I20260928 15:47:46.429629 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:46.428750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:52687)
0928 15:47:46.429323 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:46.429420 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":290,"threads_started":1}
I20260928 15:47:46.488646 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:46.487726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:40063)
0928 15:47:46.488363 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:46.488476 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":372,"threads_started":1}
I20260928 15:47:46.957481 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:46.956606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:51787)
0928 15:47:46.957115 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:46.957223 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":278,"threads_started":1}
I20260928 15:47:47.431342 28795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:47.430583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:40757)
0928 15:47:47.431069 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:47.431165 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":248,"threads_started":1}
I20260928 15:47:47.490674 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:47.489711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:43361)
0928 15:47:47.490282 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:47.490381 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
I20260928 15:47:47.959486 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:47.958589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:36633)
0928 15:47:47.959158 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:47.959248 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":245,"threads_started":1}
I20260928 15:47:48.433379 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:48.432439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:58647)
0928 15:47:48.433045 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:48.433146 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":270,"threads_started":1}
I20260928 15:47:48.492614 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:48.491788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:49189)
0928 15:47:48.492331 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:48.492420 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":327,"threads_started":1}
I20260928 15:47:48.961304 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:48.960562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:34769)
0928 15:47:48.961037 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:48.961122 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":239,"threads_started":1}
I20260928 15:47:49.435066 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:49.434348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:37139)
0928 15:47:49.434813 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:49.434902 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260928 15:47:49.494571 28802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:49.493713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:49433)
0928 15:47:49.494266 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:49.494376 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":260,"threads_started":1}
I20260928 15:47:49.963233 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:49.962421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:60151)
0928 15:47:49.962943 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:49.963035 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":236,"threads_started":1}
I20260928 15:47:50.437042 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:50.436226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:53531)
0928 15:47:50.436744 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:50.436854 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
I20260928 15:47:50.496363 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:50.495574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:52973)
0928 15:47:50.496104 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:50.496189 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":304,"threads_started":1}
I20260928 15:47:50.965183 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:50.964442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:60685)
0928 15:47:50.964921 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:50.965007 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":254,"threads_started":1}
I20260928 15:47:51.438894 28807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:51.438051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:59531)
0928 15:47:51.438611 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:51.438721 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":230,"threads_started":1}
I20260928 15:47:51.498275 28808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:51.497416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:34047)
0928 15:47:51.497945 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:51.498072 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
I20260928 15:47:51.966951 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:51.966204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:42127)
0928 15:47:51.966659 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:51.966756 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":215,"threads_started":1}
I20260928 15:47:52.440768 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:52.440021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:53035)
0928 15:47:52.440482 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:52.440582 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
I20260928 15:47:52.500259 28811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:52.499355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:34065)
0928 15:47:52.500011 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:52.500097 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":388,"threads_started":1}
I20260928 15:47:52.969537 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:52.968451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:50227)
0928 15:47:52.969035 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:52.969360 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
I20260928 15:47:53.442461 28813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:53.441759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:34705)
0928 15:47:53.442198 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:53.442280 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260928 15:47:53.502074 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:53.501221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:34557)
0928 15:47:53.501731 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:53.501843 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":271,"threads_started":1}
I20260928 15:47:53.971846 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:53.970791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:52011)
0928 15:47:53.971425 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:53.971577 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":321,"threads_started":1}
I20260928 15:47:54.444383 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:54.443504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:57587)
0928 15:47:54.444083 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:54.444189 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":232,"threads_started":1}
I20260928 15:47:54.503700 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:54.502924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:42171)
0928 15:47:54.503426 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:54.503511 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
I20260928 15:47:54.973795 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:54.972971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:57999)
0928 15:47:54.973441 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:54.973584 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260928 15:47:55.446126 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:55.445319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:55227)
0928 15:47:55.445804 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:55.445916 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":245,"threads_started":1}
I20260928 15:47:55.505754 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:55.504972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:34977)
0928 15:47:55.505436 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:55.505566 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260928 15:47:55.976318 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:55.975177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.66:32895)
0928 15:47:55.975954 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:55.976088 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":377,"threads_started":1}
I20260928 15:47:56.418020 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 28346
I20260928 15:47:56.438213 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:56.419836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.13.5.65:43301)
0928 15:47:56.420613 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:56.420632 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:56.420689 (+ 57us) server_negotiation.cc:407] Connection header received
0928 15:47:56.422533 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:56.422543 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:56.423018 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:56.423335 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:56.425031 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:56.426338 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:56.427777 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:56.428630 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:56.428777 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:56.431579 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:56.431587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:56.431603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:56.431643 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:56.434826 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:56.435538 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:56.435544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:56.435551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:56.435716 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:56.436158 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:56.436161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:56.436163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:56.436740 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:56.437021 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:56.437467 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 15:47:56.437857 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":373,"threads_started":1}
I20260928 15:47:56.439911 28505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:56.448266 28824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:56.447293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.67:32787)
0928 15:47:56.447960 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:56.448074 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":380,"threads_started":1}
I20260928 15:47:56.531266 28825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:56.529912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:38275 (local address 127.13.5.65:38251)
0928 15:47:56.530647 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:56.530873 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:38275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":475,"threads_started":1}
I20260928 15:47:56.531847 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 28346
I20260928 15:47:56.550225 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 28513
I20260928 15:47:56.569183 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:56.551245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.13.5.66:43651)
0928 15:47:56.551714 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:56.551729 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:56.551767 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:47:56.553339 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:56.553347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:56.553789 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:56.554091 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:56.555409 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:56.556858 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:56.558384 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:56.559108 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:56.559250 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:56.562484 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:56.562492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:56.562510 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:56.562553 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:56.565770 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:56.566577 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:56.566583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:56.566590 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:56.566760 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:56.567158 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:56.567161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:56.567163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:56.567718 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:56.568069 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:56.568529 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 15:47:56.568876 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":235,"threads_started":1}
I20260928 15:47:56.570750 28637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:56.658412 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 28513
I20260928 15:47:56.676429 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 28645
I20260928 15:47:56.696213 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:56.677427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.13.5.67:36713)
0928 15:47:56.678135 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:56.678150 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:56.678197 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:47:56.680084 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:56.680096 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:56.680574 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:56.680886 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:56.682286 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:56.683697 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:56.685103 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:56.685884 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:56.686041 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:56.689173 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:56.689182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:56.689200 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:56.689244 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:56.692614 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:56.693380 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:56.693387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:56.693394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:56.693638 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:56.694025 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:56.694029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:56.694031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:56.694626 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:56.694870 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:56.695291 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 15:47:56.695820 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":329,"threads_started":1}
I20260928 15:47:56.697719 28769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:47:56.785681 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 28645
2026-09-28T15:47:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24716 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:47:56 dist-test-slave-j968 krb5kdc[28832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:47:56 dist-test-slave-j968 krb5kdc[28832](info): set up 2 sockets
Sep 28 15:47:56 dist-test-slave-j968 krb5kdc[28832](info): commencing operation
krb5kdc: starting...
W20260928 15:47:57.906837 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:47:58 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:47:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:47:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:47:58.165267 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45959
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:58.503437 28845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:58.503803 28845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:58.503985 28845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:58.511215 28845 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:47:58.511359 28845 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:47:58.511461 28845 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:47:58.511572 28845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:47:58.511657 28845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:58.511758 28845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:58.511847 28845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:47:58.511950 28845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:47:58.521996 28845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:58.523367 28845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:58.524745 28845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:58.537441 28853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:58.537482 28850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:58.537441 28851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:58.538308 28845 server_base.cc:1034] running on GCE node
I20260928 15:47:58.539179 28845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:58.540668 28845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:58.541889 28845 hybrid_clock.cc:648] HybridClock initialized: now 1790610478541835 us; error 27 us; skew 500 ppm
I20260928 15:47:58.542191 28845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:58 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610478, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:58.549580 28845 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:47:58.551322 28845 webserver.cc:460] Webserver started at http://127.13.5.126:38459/ using document root <none> and password file <none>
I20260928 15:47:58.552152 28845 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:58.552253 28845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:58.552561 28845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:58.556149 28845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "0ada969f3d5f4076bd5233a91d976513"
format_stamp: "Formatted at 2026-09-28 15:47:58 on dist-test-slave-j968"
server_key: "38859c7e2a1674a0a4040008b0d06fb8"
server_key_iv: "b9f32f805f2315a3185ab0fb35d00c89"
server_key_version: "encryptionkey@0"
I20260928 15:47:58.557281 28845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "0ada969f3d5f4076bd5233a91d976513"
format_stamp: "Formatted at 2026-09-28 15:47:58 on dist-test-slave-j968"
server_key: "38859c7e2a1674a0a4040008b0d06fb8"
server_key_iv: "b9f32f805f2315a3185ab0fb35d00c89"
server_key_version: "encryptionkey@0"
I20260928 15:47:58.563148 28845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:47:58.566855 28860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:58.568367 28845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:47:58.568560 28845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "0ada969f3d5f4076bd5233a91d976513"
format_stamp: "Formatted at 2026-09-28 15:47:58 on dist-test-slave-j968"
server_key: "38859c7e2a1674a0a4040008b0d06fb8"
server_key_iv: "b9f32f805f2315a3185ab0fb35d00c89"
server_key_version: "encryptionkey@0"
I20260928 15:47:58.568847 28845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:58.598834 28845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:58.603165 28845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:58.603464 28845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:58.622941 28845 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:45959
I20260928 15:47:58.622988 28911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:45959 every 8 connection(s)
I20260928 15:47:58.624637 28845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:47:58.629215 28912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:47:58.629967 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 28845
I20260928 15:47:58.630203 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:47:58.630884 13333 external_mini_cluster.cc:1428] Setting key 12afb654003c5e8a8e2e2a229afa4592
I20260928 15:47:58.644506 28912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Bootstrap starting.
Sep 28 15:47:58 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:58.648633 28912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Neither blocks nor log segments found. Creating new log.
I20260928 15:47:58.650236 28912 log.cc:824] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Log is configured to *not* fsync() on all Append() calls
I20260928 15:47:58.653919 28912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: No bootstrap required, opened a new log
I20260928 15:47:58.656662 28915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:58.633097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.13.5.126:45959)
0928 15:47:58.633933 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:58.633945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:58.633986 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:47:58.635621 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:58.635647 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:58.636126 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:58.636484 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:58.638358 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:58.639991 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:58.642018 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:58.642745 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:58.642954 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:58.647843 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:58.647854 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:58.647901 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:58.647954 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:58.651843 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:58.652571 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:58.652580 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:58.652589 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:58.652802 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:58.653204 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:58.653209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:58.653212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:58.653937 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:58.654231 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:58.654710 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 15:47:58.655183 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":371,"threads_started":1}
I20260928 15:47:58.663399 28912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:47:58.663772 28912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:47:58.663849 28912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ada969f3d5f4076bd5233a91d976513, State: Initialized, Role: FOLLOWER
I20260928 15:47:58.664425 28912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:47:58.664637 28912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:47:58.664737 28912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:47:58.664861 28912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:47:58.667196 28912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:47:58.667773 28912 leader_election.cc:302] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ada969f3d5f4076bd5233a91d976513; no voters:
I20260928 15:47:58.668419 28912 leader_election.cc:288] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:47:58.668584 28917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:47:58.669294 28917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 1 LEADER]: Becoming Leader. State: Replica: 0ada969f3d5f4076bd5233a91d976513, State: Running, Role: LEADER
I20260928 15:47:58.669883 28917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:47:58.670589 28912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:47:58.673178 28919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ada969f3d5f4076bd5233a91d976513. Latest consensus state: current_term: 1 leader_uuid: "0ada969f3d5f4076bd5233a91d976513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } } }
I20260928 15:47:58.673178 28918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ada969f3d5f4076bd5233a91d976513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } } }
I20260928 15:47:58.673648 28919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:58.673727 28918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: This master's current role is: LEADER
I20260928 15:47:58.676151 28922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:47:58.682442 28922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:47:58.690714 28922 catalog_manager.cc:1347] Generated new cluster ID: 1c360593c2d74f1fb237a906ced4cd74
I20260928 15:47:58.690834 28922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:47:58.704324 28922 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:47:58.706271 28922 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:47:58.712759 28922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Generated new TSK 0
I20260928 15:47:58.713822 28922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:47:58.784168 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45959
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:59.135515 28938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:59.135845 28938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:59.136044 28938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:59.143451 28938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:59.143616 28938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:59.143785 28938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:47:59.153877 28938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:59.155284 28938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:59.156881 28938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:59.170814 28946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:59.171011 28943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:59.170755 28944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:59.172313 28938 server_base.cc:1034] running on GCE node
I20260928 15:47:59.173079 28938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:59.174129 28938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:59.175396 28938 hybrid_clock.cc:648] HybridClock initialized: now 1790610479175333 us; error 42 us; skew 500 ppm
I20260928 15:47:59.175757 28938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:59 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610479, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:59.179208 28938 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:47:59.180754 28938 webserver.cc:460] Webserver started at http://127.13.5.65:34239/ using document root <none> and password file <none>
I20260928 15:47:59.181494 28938 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:59.181648 28938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:59.181979 28938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:59.185050 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "eb10e06365564c23bf1b422f34a2c03f"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "221c8805f3cb13e50d57992e0786bbdc"
server_key_iv: "e2773694141c71798363db1b8da3f786"
server_key_version: "encryptionkey@0"
I20260928 15:47:59.185817 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "eb10e06365564c23bf1b422f34a2c03f"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "221c8805f3cb13e50d57992e0786bbdc"
server_key_iv: "e2773694141c71798363db1b8da3f786"
server_key_version: "encryptionkey@0"
I20260928 15:47:59.191191 28938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 15:47:59.194922 28954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:59.196475 28938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:47:59.196674 28938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "eb10e06365564c23bf1b422f34a2c03f"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "221c8805f3cb13e50d57992e0786bbdc"
server_key_iv: "e2773694141c71798363db1b8da3f786"
server_key_version: "encryptionkey@0"
I20260928 15:47:59.196900 28938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:59.233994 28938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:59.238660 28938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:59.238981 28938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:59.240204 28938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:59.241585 28938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:59.241668 28938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:59.241844 28938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:59.241905 28938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:59.275447 28938 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:41567
I20260928 15:47:59.275542 29068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:41567 every 8 connection(s)
I20260928 15:47:59.276949 28938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:47:59.283767 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 28938
I20260928 15:47:59.284017 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:47:59.284693 13333 external_mini_cluster.cc:1428] Setting key 0836a22fd9e139cf277db3042dac91f6
Sep 28 15:47:59 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610479, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:59.299309 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:59.279675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:43767 (local address 127.13.5.126:45959)
0928 15:47:59.280204 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:59.280208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:59.281061 (+ 853us) server_negotiation.cc:407] Connection header received
0928 15:47:59.283006 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:59.283011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:59.283110 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:59.283232 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:59.285001 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:59.285956 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:59.288311 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:59.288706 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:59.288809 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:59.293331 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:59.293337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:59.293340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:59.293374 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:59.296127 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:59.296911 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:59.296916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:59.296918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:59.297045 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:59.297497 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:59.297508 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:59.297511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:59.297774 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:59.297932 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:59.298691 (+ 759us) server_negotiation.cc:299] Negotiation successful
0928 15:47:59.298928 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":310,"threads_started":1}
I20260928 15:47:59.300441 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:59.280197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:43767)
0928 15:47:59.280894 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:59.280933 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:59.282712 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:59.283495 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:59.283504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:59.284171 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:59.284810 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:59.284837 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:59.286129 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:59.286133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:59.288159 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:59.288178 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:59.288949 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:59.288959 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:59.289286 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:59.290163 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:59.290197 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:59.293039 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:59.296287 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:59.296296 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:59.296316 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:59.296766 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:59.297162 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:59.297166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:59.297170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:59.297374 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:59.297917 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:59.297926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:59.298277 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 15:47:59.298566 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 15:47:59.299125 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":284,"threads_started":1}
I20260928 15:47:59.302222 29069 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45959
I20260928 15:47:59.302556 29069 heartbeater.cc:464] Registering TS with master...
I20260928 15:47:59.303306 29069 heartbeater.cc:511] Master 127.13.5.126:45959 requested a full tablet report, sending...
I20260928 15:47:59.305725 28877 ts_manager.cc:194] Registered new tserver with Master: eb10e06365564c23bf1b422f34a2c03f (127.13.5.65:41567)
I20260928 15:47:59.308153 28877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:43767
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:47:59.344162 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45959
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:47:59.680553 29075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:47:59.680917 29075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:47:59.681089 29075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:47:59.688656 29075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:47:59.688812 29075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:47:59.689009 29075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:47:59.699366 29075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:47:59.700959 29075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:47:59.702433 29075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:47:59.717432 29081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:59.717531 29083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:47:59.717482 29080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:47:59.718801 29075 server_base.cc:1034] running on GCE node
I20260928 15:47:59.719494 29075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:47:59.720539 29075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:47:59.721751 29075 hybrid_clock.cc:648] HybridClock initialized: now 1790610479721707 us; error 31 us; skew 500 ppm
I20260928 15:47:59.722043 29075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:47:59 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610479, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:47:59.725569 29075 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:47:59.727069 29075 webserver.cc:460] Webserver started at http://127.13.5.66:38825/ using document root <none> and password file <none>
I20260928 15:47:59.727800 29075 fs_manager.cc:359] Metadata directory not provided
I20260928 15:47:59.727995 29075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:47:59.728332 29075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:47:59.731329 29075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "ad4bd7711d9a4b638edc27ccabd81875"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "15a809aab2d976b9e33ddbefc9aedbf6"
server_key_iv: "032f6bdcc28ad81fb8e404ea7a3ccd8f"
server_key_version: "encryptionkey@0"
I20260928 15:47:59.732196 29075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "ad4bd7711d9a4b638edc27ccabd81875"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "15a809aab2d976b9e33ddbefc9aedbf6"
server_key_iv: "032f6bdcc28ad81fb8e404ea7a3ccd8f"
server_key_version: "encryptionkey@0"
I20260928 15:47:59.737651 29075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:47:59.741320 29090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:59.742754 29075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:47:59.742933 29075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "ad4bd7711d9a4b638edc27ccabd81875"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "15a809aab2d976b9e33ddbefc9aedbf6"
server_key_iv: "032f6bdcc28ad81fb8e404ea7a3ccd8f"
server_key_version: "encryptionkey@0"
I20260928 15:47:59.743196 29075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:47:59.762741 29075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:47:59.767547 29075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:47:59.767911 29075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:47:59.768950 29075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:47:59.770434 29075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:47:59.770509 29075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:59.770684 29075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:47:59.770732 29075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:47:59.806854 29075 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:33071
I20260928 15:47:59.806900 29203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:33071 every 8 connection(s)
I20260928 15:47:59.808290 29075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:47:59.809751 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 29075
I20260928 15:47:59.809948 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:47:59.810405 13333 external_mini_cluster.cc:1428] Setting key 3f82238098f35c93c917f1c5e384f1dc
Sep 28 15:47:59 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610479, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:47:59.831382 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:59.811085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:48775 (local address 127.13.5.126:45959)
0928 15:47:59.812297 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0928 15:47:59.812303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:47:59.813009 (+ 706us) server_negotiation.cc:407] Connection header received
0928 15:47:59.814834 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:47:59.814842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:47:59.814932 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:47:59.815051 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:47:59.816696 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:59.817965 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:59.819592 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:47:59.820250 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:47:59.820399 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:59.824511 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:47:59.824517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:47:59.824520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:47:59.824589 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:47:59.827419 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:59.828374 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:59.828380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:59.828383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:59.828530 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:47:59.829113 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:47:59.829117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:47:59.829137 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:47:59.829434 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:47:59.829611 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 15:47:59.830559 (+ 948us) server_negotiation.cc:299] Negotiation successful
0928 15:47:59.830835 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":920,"threads_started":1}
I20260928 15:47:59.832118 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:47:59.811751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:48775)
0928 15:47:59.812832 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0928 15:47:59.812878 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:47:59.814554 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:47:59.815258 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:47:59.815266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:47:59.815811 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:47:59.816510 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:59.816528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:59.818146 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:59.818150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:59.819431 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:47:59.819443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:47:59.820522 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:47:59.820530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:47:59.820785 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:47:59.821532 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:47:59.821614 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:47:59.824227 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:47:59.827609 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:59.827619 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:59.827640 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:59.828222 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:59.828657 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:47:59.828662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:47:59.828666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:47:59.828967 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:47:59.829583 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:47:59.829593 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:47:59.830047 (+ 454us) client_negotiation.cc:769] Sending connection context
0928 15:47:59.830393 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 15:47:59.830851 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":449,"threads_started":1}
I20260928 15:47:59.833791 29204 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45959
I20260928 15:47:59.834177 29204 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
I20260928 15:47:59.835171 29204 heartbeater.cc:511] Master 127.13.5.126:45959 requested a full tablet report, sending...
I20260928 15:47:59.836807 28877 ts_manager.cc:194] Registered new tserver with Master: ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
I20260928 15:47:59.837661 28877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:48775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:47:59.868067 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45959
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:48:00.205416 29210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:00.205724 29210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:00.205818 29210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:00.213219 29210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:00.213337 29210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:00.213454 29210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:48:00.223412 29210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:00.224779 29210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:00.226238 29210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:00.239915 29218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:00.239979 29216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:00.240177 29215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:00.240656 29210 server_base.cc:1034] running on GCE node
I20260928 15:48:00.241312 29210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:00.242322 29210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:00.243522 29210 hybrid_clock.cc:648] HybridClock initialized: now 1790610480243476 us; error 33 us; skew 500 ppm
I20260928 15:48:00.243853 29210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:00 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610480, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:00.247205 29210 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:48:00.248823 29210 webserver.cc:460] Webserver started at http://127.13.5.67:35671/ using document root <none> and password file <none>
I20260928 15:48:00.249527 29210 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:00.249676 29210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:00.250008 29210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:48:00.253106 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "9e2729c54e3b43ceaa01a9c6926d55b9"
format_stamp: "Formatted at 2026-09-28 15:48:00 on dist-test-slave-j968"
server_key: "ebb703d9ab5f385b5d54fa5e754e1920"
server_key_iv: "a77c84c5e24e0920657c04468b48e7ac"
server_key_version: "encryptionkey@0"
I20260928 15:48:00.253877 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "9e2729c54e3b43ceaa01a9c6926d55b9"
format_stamp: "Formatted at 2026-09-28 15:48:00 on dist-test-slave-j968"
server_key: "ebb703d9ab5f385b5d54fa5e754e1920"
server_key_iv: "a77c84c5e24e0920657c04468b48e7ac"
server_key_version: "encryptionkey@0"
I20260928 15:48:00.259163 29210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:48:00.262717 29225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:00.264153 29210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:48:00.264317 29210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "9e2729c54e3b43ceaa01a9c6926d55b9"
format_stamp: "Formatted at 2026-09-28 15:48:00 on dist-test-slave-j968"
server_key: "ebb703d9ab5f385b5d54fa5e754e1920"
server_key_iv: "a77c84c5e24e0920657c04468b48e7ac"
server_key_version: "encryptionkey@0"
I20260928 15:48:00.264530 29210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:00.285805 29210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:00.290306 29210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:00.290618 29210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:00.291744 29210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:48:00.293192 29210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:48:00.293268 29210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:00.293401 29210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:48:00.293450 29210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:00.311616 29069 heartbeater.cc:503] Master 127.13.5.126:45959 was elected leader, sending a full tablet report...
I20260928 15:48:00.327687 29210 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:34569
I20260928 15:48:00.327792 29338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:34569 every 8 connection(s)
I20260928 15:48:00.329176 29210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:48:00.332386 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 29210
I20260928 15:48:00.332667 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:48:00.333330 13333 external_mini_cluster.cc:1428] Setting key c19d29f381751271777ed0745f64330a
Sep 28 15:48:00 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610480, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:48:00.350811 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.332081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:53311 (local address 127.13.5.126:45959)
0928 15:48:00.332608 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.332616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.333696 (+ 1080us) server_negotiation.cc:407] Connection header received
0928 15:48:00.336286 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.336291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.336375 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.336501 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:00.338129 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.338896 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.340517 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.340917 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.341014 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.344459 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:00.344464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:00.344467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:00.344515 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:00.347349 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:00.348406 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:00.348410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:00.348412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:00.348527 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:00.349099 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:00.349104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:00.349106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:00.349505 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:00.349664 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.350367 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.350533 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":238,"threads_started":1}
I20260928 15:48:00.351787 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.332593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:53311)
0928 15:48:00.333499 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:00.333564 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:00.335984 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:00.336777 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:00.336789 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:00.337392 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:48:00.337960 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.337988 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.339038 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.339043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.340383 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.340396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.341121 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.341126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.341317 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.341952 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:48:00.341997 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:48:00.344235 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:48:00.347492 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:00.347501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:00.347546 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:00.348279 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:00.348664 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:00.348667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:00.348670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:00.348937 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:00.349647 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:48:00.349655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:48:00.350031 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 15:48:00.350260 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 15:48:00.350656 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":320,"threads_started":1}
I20260928 15:48:00.353159 29339 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45959
I20260928 15:48:00.353492 29339 heartbeater.cc:464] Registering TS with master...
I20260928 15:48:00.354242 29339 heartbeater.cc:511] Master 127.13.5.126:45959 requested a full tablet report, sending...
I20260928 15:48:00.355856 28877 ts_manager.cc:194] Registered new tserver with Master: 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569)
I20260928 15:48:00.356719 28877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:53311
I20260928 15:48:00.362032 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:48:00.380131 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.366757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.13.5.126:45959)
0928 15:48:00.367006 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.367013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.367484 (+ 471us) server_negotiation.cc:407] Connection header received
0928 15:48:00.367627 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.367631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.367708 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.367792 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:00.369371 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.370115 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.371825 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.372311 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.372431 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.374829 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:00.374834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:00.374837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:00.374869 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:00.377653 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:00.378379 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:00.378382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:00.378385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:00.378508 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:00.378821 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:00.378824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:00.378826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:00.379156 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:00.379347 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.379633 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.379806 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:48:00.384742 28877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:48:00.393035 28877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:48:00.425125 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.406145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.13.5.65:41567)
0928 15:48:00.413000 (+ 6855us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.413013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.414239 (+ 1226us) server_negotiation.cc:407] Connection header received
0928 15:48:00.414404 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.414416 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.414785 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.415031 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:00.416287 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.418485 (+ 2198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.421929 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.423470 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.423602 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.423845 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.424572 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.424758 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6620,"thread_start_us":6441,"threads_started":1}
I20260928 15:48:00.425125 29352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.405890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:42288)
0928 15:48:00.413627 (+ 7737us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:00.413687 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:00.413966 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:00.415432 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:00.415443 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:00.415501 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:00.416038 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.416054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.418707 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.418716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.421714 (+ 2998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.421756 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.424017 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.424025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.424266 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.424321 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 15:48:00.424455 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 15:48:00.424681 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7361,"thread_start_us":7223,"threads_started":1}
I20260928 15:48:00.429083 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.406171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:34569 (local address 127.0.0.1:54356)
0928 15:48:00.420822 (+ 14651us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:00.420846 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:00.421001 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:00.421799 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:00.421806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:00.421841 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:00.422302 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.422318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.423879 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.423914 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.426130 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.426145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.428288 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.428294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.428490 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.428525 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:48:00.428612 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:48:00.428747 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14494,"thread_start_us":14684,"threads_started":1}
I20260928 15:48:00.429283 29003 tablet_service.cc:1467] Processing CreateTablet for tablet d8fcdf92fc484db79feed8dfba85ab87 (DEFAULT_TABLE table=test-table [id=545779505584413a943b0b828fe71b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:48:00.429891 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.406790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.13.5.67:34569)
0928 15:48:00.419603 (+ 12813us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.419616 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.421139 (+ 1523us) server_negotiation.cc:407] Connection header received
0928 15:48:00.421256 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.421265 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.421594 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.421798 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:00.422559 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.423714 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.426308 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.427747 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.427904 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.428153 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.429375 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.429555 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12579,"thread_start_us":12839,"threads_started":1}
I20260928 15:48:00.430915 29003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fcdf92fc484db79feed8dfba85ab87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:48:00.431110 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.406996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.13.5.66:33071)
0928 15:48:00.420295 (+ 13299us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.420307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.421393 (+ 1086us) server_negotiation.cc:407] Connection header received
0928 15:48:00.421496 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.421504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.421719 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.421887 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:00.422754 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.425072 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.428401 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.429872 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.430027 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.430222 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.430670 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.430833 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13149,"thread_start_us":13031,"threads_started":1}
I20260928 15:48:00.433682 29274 tablet_service.cc:1467] Processing CreateTablet for tablet d8fcdf92fc484db79feed8dfba85ab87 (DEFAULT_TABLE table=test-table [id=545779505584413a943b0b828fe71b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:48:00.434922 29274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fcdf92fc484db79feed8dfba85ab87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:48:00.431048 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.406496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33071 (local address 127.0.0.1:37776)
0928 15:48:00.421129 (+ 14633us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:00.421147 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:00.421255 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:00.422130 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:00.422134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:00.422158 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:00.422574 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.422586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.425482 (+ 2896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.425488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.427821 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.427837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.430350 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.430355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.430471 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.430496 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 15:48:00.430606 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:48:00.430732 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14490,"thread_start_us":13998,"threads_started":1}
I20260928 15:48:00.441421 29139 tablet_service.cc:1467] Processing CreateTablet for tablet d8fcdf92fc484db79feed8dfba85ab87 (DEFAULT_TABLE table=test-table [id=545779505584413a943b0b828fe71b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:48:00.442986 29139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fcdf92fc484db79feed8dfba85ab87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:48:00.445330 29358 tablet_bootstrap.cc:492] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Bootstrap starting.
I20260928 15:48:00.447429 29359 tablet_bootstrap.cc:492] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Bootstrap starting.
I20260928 15:48:00.449311 29358 tablet_bootstrap.cc:654] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Neither blocks nor log segments found. Creating new log.
I20260928 15:48:00.450459 29358 log.cc:824] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:00.451125 29359 tablet_bootstrap.cc:654] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Neither blocks nor log segments found. Creating new log.
I20260928 15:48:00.452455 29359 log.cc:824] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:00.454216 29358 tablet_bootstrap.cc:492] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: No bootstrap required, opened a new log
I20260928 15:48:00.454500 29358 ts_tablet_manager.cc:1397] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260928 15:48:00.455469 29359 tablet_bootstrap.cc:492] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: No bootstrap required, opened a new log
I20260928 15:48:00.455722 29359 ts_tablet_manager.cc:1397] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 15:48:00.457880 29362 tablet_bootstrap.cc:492] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Bootstrap starting.
I20260928 15:48:00.461969 29362 tablet_bootstrap.cc:654] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Neither blocks nor log segments found. Creating new log.
I20260928 15:48:00.463258 29362 log.cc:824] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:00.464143 29358 raft_consensus.cc:348] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.464530 29358 raft_consensus.cc:374] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:48:00.464682 29358 raft_consensus.cc:729] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb10e06365564c23bf1b422f34a2c03f, State: Initialized, Role: FOLLOWER
I20260928 15:48:00.464448 29359 raft_consensus.cc:348] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.464897 29359 raft_consensus.cc:374] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:48:00.465148 29359 raft_consensus.cc:729] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2729c54e3b43ceaa01a9c6926d55b9, State: Initialized, Role: FOLLOWER
I20260928 15:48:00.465415 29358 consensus_queue.cc:260] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.465813 29359 consensus_queue.cc:260] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.467278 29358 ts_tablet_manager.cc:1428] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260928 15:48:00.468142 29362 tablet_bootstrap.cc:492] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: No bootstrap required, opened a new log
I20260928 15:48:00.468508 29362 ts_tablet_manager.cc:1397] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 15:48:00.468619 29339 heartbeater.cc:503] Master 127.13.5.126:45959 was elected leader, sending a full tablet report...
I20260928 15:48:00.469255 29359 ts_tablet_manager.cc:1428] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260928 15:48:00.477305 29362 raft_consensus.cc:348] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.477723 29362 raft_consensus.cc:374] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:48:00.477801 29362 raft_consensus.cc:729] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4bd7711d9a4b638edc27ccabd81875, State: Initialized, Role: FOLLOWER
I20260928 15:48:00.478299 29362 consensus_queue.cc:260] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.479247 29204 heartbeater.cc:503] Master 127.13.5.126:45959 was elected leader, sending a full tablet report...
I20260928 15:48:00.479724 29362 ts_tablet_manager.cc:1428] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260928 15:48:00.529681 29070 tablet.cc:2369] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:48:00.560470 29205 tablet.cc:2369] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:48:00.580926 29340 tablet.cc:2369] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:48:00.646193 29365 raft_consensus.cc:482] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:48:00.646512 29365 raft_consensus.cc:504] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.648367 29365 leader_election.cc:288] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb10e06365564c23bf1b422f34a2c03f (127.13.5.65:41567), ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
I20260928 15:48:00.655062 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.648770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:36475 (local address 127.13.5.65:41567)
0928 15:48:00.649112 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.649121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.649142 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:48:00.649227 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.649235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.649325 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.649424 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:00.650033 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.651131 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.653029 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.654101 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.654203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.654282 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.654692 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.654789 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260928 15:48:00.655133 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.648651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.13.5.67:36475)
0928 15:48:00.648884 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:00.648904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:00.649045 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:00.649585 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:00.649591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:00.649627 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:00.649833 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.649869 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.651284 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.651289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.652880 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.652893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.654281 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.654288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.654496 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.654540 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 15:48:00.654647 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:48:00.654792 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":22}
I20260928 15:48:00.655948 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.649311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:55043 (local address 127.13.5.66:33071)
0928 15:48:00.649560 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.649566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.649787 (+ 221us) server_negotiation.cc:407] Connection header received
0928 15:48:00.649931 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.649939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.650037 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.650105 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:00.650770 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.651713 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.653794 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.654895 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.655030 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.655124 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.655524 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.655660 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 15:48:00.656056 29367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.648872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33071 (local address 127.13.5.67:55043)
0928 15:48:00.649678 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:00.649693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:00.649803 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:00.650169 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:00.650174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:00.650200 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:00.650620 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.650633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.651914 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.651922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.653613 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:00.653627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.655125 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.655135 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:00.655338 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.655369 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 15:48:00.655485 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:48:00.655659 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":44,"thread_start_us":404,"threads_started":1}
I20260928 15:48:00.656452 29023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fcdf92fc484db79feed8dfba85ab87" candidate_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb10e06365564c23bf1b422f34a2c03f" is_pre_election: true
I20260928 15:48:00.657128 29023 raft_consensus.cc:2454] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2729c54e3b43ceaa01a9c6926d55b9 in term 0.
I20260928 15:48:00.657229 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fcdf92fc484db79feed8dfba85ab87" candidate_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4bd7711d9a4b638edc27ccabd81875" is_pre_election: true
I20260928 15:48:00.657774 29159 raft_consensus.cc:2454] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2729c54e3b43ceaa01a9c6926d55b9 in term 0.
I20260928 15:48:00.657938 29227 leader_election.cc:302] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e2729c54e3b43ceaa01a9c6926d55b9, eb10e06365564c23bf1b422f34a2c03f; no voters:
I20260928 15:48:00.658456 29365 raft_consensus.cc:2781] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:48:00.658685 29365 raft_consensus.cc:482] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:48:00.658813 29365 raft_consensus.cc:3037] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:48:00.661139 29365 raft_consensus.cc:504] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.661791 29365 leader_election.cc:288] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [CANDIDATE]: Term 1 election: Requested vote from peers eb10e06365564c23bf1b422f34a2c03f (127.13.5.65:41567), ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
I20260928 15:48:00.662328 29023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fcdf92fc484db79feed8dfba85ab87" candidate_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb10e06365564c23bf1b422f34a2c03f"
I20260928 15:48:00.662328 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fcdf92fc484db79feed8dfba85ab87" candidate_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4bd7711d9a4b638edc27ccabd81875"
I20260928 15:48:00.662549 29023 raft_consensus.cc:3037] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:48:00.662760 29159 raft_consensus.cc:3037] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:48:00.666203 29159 raft_consensus.cc:2454] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2729c54e3b43ceaa01a9c6926d55b9 in term 1.
I20260928 15:48:00.666203 29023 raft_consensus.cc:2454] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2729c54e3b43ceaa01a9c6926d55b9 in term 1.
I20260928 15:48:00.666766 29227 leader_election.cc:302] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e2729c54e3b43ceaa01a9c6926d55b9, eb10e06365564c23bf1b422f34a2c03f; no voters:
I20260928 15:48:00.667099 29365 raft_consensus.cc:2781] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:48:00.667644 29365 raft_consensus.cc:686] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2729c54e3b43ceaa01a9c6926d55b9, State: Running, Role: LEADER
I20260928 15:48:00.668239 29365 consensus_queue.cc:237] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:00.673146 28876 catalog_manager.cc:5495] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } health_report { overall_health: UNKNOWN } } }
I20260928 15:48:00.729996 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.724518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.13.5.67:34569)
0928 15:48:00.724739 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.724744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.724762 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:48:00.724953 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.724959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.725053 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.725192 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:48:00.725545 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.726285 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.727815 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.728263 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.728352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.728926 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:48:00.729475 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:48:00.729587 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.729673 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.729743 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:48:00.738775 29159 raft_consensus.cc:1261] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 1 FOLLOWER]: Refusing update from remote peer 9e2729c54e3b43ceaa01a9c6926d55b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:48:00.738775 29023 raft_consensus.cc:1261] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Refusing update from remote peer 9e2729c54e3b43ceaa01a9c6926d55b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:48:00.739809 29368 consensus_queue.cc:1035] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:48:00.740247 29365 consensus_queue.cc:1035] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:48:00.754136 29373 mvcc.cc:205] Tried to move back new op lower bound from 7334340529095864320 to 7334340528821284864. Current Snapshot: MvccSnapshot[applied={T|T < 7334340529095864320}]
I20260928 15:48:00.769883 28876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:48:00.770275 28876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:48:00.774227 28876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Sending DeleteTablet for 3 replicas of tablet d8fcdf92fc484db79feed8dfba85ab87
I20260928 15:48:00.775452 29274 tablet_service.cc:1514] Processing DeleteTablet for tablet d8fcdf92fc484db79feed8dfba85ab87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:48:00 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:54356
I20260928 15:48:00.775509 29139 tablet_service.cc:1514] Processing DeleteTablet for tablet d8fcdf92fc484db79feed8dfba85ab87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:48:00 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:37776
I20260928 15:48:00.776221 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 28938
I20260928 15:48:00.776209 29003 tablet_service.cc:1514] Processing DeleteTablet for tablet d8fcdf92fc484db79feed8dfba85ab87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:48:00 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:42288
I20260928 15:48:00.785513 29382 tablet_replica.cc:330] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: stopping tablet replica
I20260928 15:48:00.787029 29382 raft_consensus.cc:2229] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:48:00.788410 29382 raft_consensus.cc:2258] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:48:00.790591 29380 tablet_replica.cc:330] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: stopping tablet replica
I20260928 15:48:00.791208 29380 raft_consensus.cc:2229] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:48:00.791806 29382 ts_tablet_manager.cc:1905] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:48:00.792019 29380 raft_consensus.cc:2258] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:48:00.794358 29380 ts_tablet_manager.cc:1905] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:48:00 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:48:00.802357 29380 ts_tablet_manager.cc:1918] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:48:00.802675 29380 log.cc:1196] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/d8fcdf92fc484db79feed8dfba85ab87
I20260928 15:48:00.803344 29380 ts_tablet_manager.cc:1939] T d8fcdf92fc484db79feed8dfba85ab87 P 9e2729c54e3b43ceaa01a9c6926d55b9: Deleting consensus metadata
I20260928 15:48:00.805482 28861 catalog_manager.cc:4841] TS 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569): tablet d8fcdf92fc484db79feed8dfba85ab87 (table test-table [id=545779505584413a943b0b828fe71b50]) successfully deleted
I20260928 15:48:00.805569 29381 tablet_replica.cc:330] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: stopping tablet replica
I20260928 15:48:00.806085 29381 raft_consensus.cc:2229] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:48:00.807937 29382 ts_tablet_manager.cc:1918] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:48:00.808202 29382 log.cc:1196] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/d8fcdf92fc484db79feed8dfba85ab87
I20260928 15:48:00.808938 29382 ts_tablet_manager.cc:1939] T d8fcdf92fc484db79feed8dfba85ab87 P eb10e06365564c23bf1b422f34a2c03f: Deleting consensus metadata
I20260928 15:48:00.810863 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.777964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.13.5.65:41567)
0928 15:48:00.778207 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.778217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.788021 (+ 9804us) server_negotiation.cc:407] Connection header received
0928 15:48:00.788569 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.788577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.788688 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.788788 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:00.791189 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.792479 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.794811 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.795491 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.795602 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.802181 (+ 6579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:00.802200 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:00.802220 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:00.802275 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:00.806689 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:00.807825 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:00.807834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:00.807837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:00.808100 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:00.808664 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:00.808672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:00.808678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:00.809195 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:00.809498 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.810189 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.810453 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:48:00.811100 28862 catalog_manager.cc:4841] TS eb10e06365564c23bf1b422f34a2c03f (127.13.5.65:41567): tablet d8fcdf92fc484db79feed8dfba85ab87 (table test-table [id=545779505584413a943b0b828fe71b50]) successfully deleted
I20260928 15:48:00.812407 29381 raft_consensus.cc:2258] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:48:00.812705 29064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:48:00.814391 29381 ts_tablet_manager.cc:1905] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:48:00.822962 29381 ts_tablet_manager.cc:1918] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:48:00.823259 29381 log.cc:1196] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/d8fcdf92fc484db79feed8dfba85ab87
I20260928 15:48:00.823838 29381 ts_tablet_manager.cc:1939] T d8fcdf92fc484db79feed8dfba85ab87 P ad4bd7711d9a4b638edc27ccabd81875: Deleting consensus metadata
I20260928 15:48:00.825960 28863 catalog_manager.cc:4841] TS ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071): tablet d8fcdf92fc484db79feed8dfba85ab87 (table test-table [id=545779505584413a943b0b828fe71b50]) successfully deleted
I20260928 15:48:00.951650 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 28938
I20260928 15:48:00.977080 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 29075
Sep 28 15:48:00 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:48:00.994381 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:00.977989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.13.5.66:33071)
0928 15:48:00.978182 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:00.978189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:00.978211 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:48:00.978354 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:00.978360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:00.978463 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:00.978575 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:00.979936 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.981109 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.982457 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:00.983093 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:00.983225 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:00.988115 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:00.988133 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:00.988151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:00.988190 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:00.991241 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:00.992200 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:00.992208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:00.992211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:00.992384 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:00.992863 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:00.992868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:00.992870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:00.993258 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:00.993578 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:00.993867 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 15:48:00.994064 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:48:00.995186 29199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:48:01.087038 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 29075
I20260928 15:48:01.105929 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 29210
Sep 28 15:48:01 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:48:01.125522 29356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:01.106977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.13.5.67:34569)
0928 15:48:01.107162 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:01.107166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:01.107185 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:48:01.107286 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:01.107290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:01.107354 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:01.107440 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:01.109460 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:01.110415 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:01.112553 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:01.112993 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:01.113103 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:01.118963 (+ 5860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:01.118980 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:01.119000 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:01.119049 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:01.122394 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:01.123386 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:01.123396 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:01.123399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:01.123571 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:01.124051 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:01.124054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:01.124056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:01.124437 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:01.124690 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:01.125064 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 15:48:01.125230 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:48:01.126441 29334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:48:01.251475 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 29210
I20260928 15:48:01.273283 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 28845
I20260928 15:48:01.274716 28907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:48:01.388113 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 28845
I20260928 15:48:21.411278 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45959
--webserver_interface=127.13.5.126
--webserver_port=38459
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:48:21.764621 29388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:21.765098 29388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:21.765276 29388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:21.772778 29388 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:48:21.772886 29388 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:48:21.772967 29388 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:48:21.773043 29388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:48:21.773138 29388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:21.773259 29388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:21.773337 29388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:48:21.773423 29388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:48:21.783902 29388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=38459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:21.785486 29388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:21.786978 29388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:21.811980 29395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:21.812276 29394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:21.812652 29397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:21.812908 29388 server_base.cc:1034] running on GCE node
I20260928 15:48:21.813910 29388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:21.815794 29388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:21.817144 29388 hybrid_clock.cc:648] HybridClock initialized: now 1790610501817077 us; error 46 us; skew 500 ppm
I20260928 15:48:21.817638 29388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:21 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610501, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:21.826339 29388 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:48:21.828605 29388 webserver.cc:460] Webserver started at http://127.13.5.126:38459/ using document root <none> and password file <none>
I20260928 15:48:21.829557 29388 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:21.829713 29388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:21.836730 29388 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:48:21.840169 29404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:21.842025 29388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:48:21.842257 29388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "0ada969f3d5f4076bd5233a91d976513"
format_stamp: "Formatted at 2026-09-28 15:47:58 on dist-test-slave-j968"
server_key: "38859c7e2a1674a0a4040008b0d06fb8"
server_key_iv: "b9f32f805f2315a3185ab0fb35d00c89"
server_key_version: "encryptionkey@0"
I20260928 15:48:21.843477 29388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:21.859941 29388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:21.865558 29388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:21.865899 29388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:21.887668 29388 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:45959
I20260928 15:48:21.887691 29456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:45959 every 8 connection(s)
I20260928 15:48:21.889503 29388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:48:21.897425 29457 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:48:21.898104 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 29388
I20260928 15:48:21.899029 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:41567
--local_ip_for_outbound_sockets=127.13.5.65
--tserver_master_addrs=127.13.5.126:45959
--webserver_port=34239
--webserver_interface=127.13.5.65
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:48:21.901291 29457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Bootstrap starting.
I20260928 15:48:21.922416 29457 log.cc:824] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:21.935849 29457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:48:21.936514 29457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Bootstrap complete.
I20260928 15:48:21.946174 29457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:48:21.947110 29457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ada969f3d5f4076bd5233a91d976513, State: Initialized, Role: FOLLOWER
I20260928 15:48:21.947937 29457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:48:21.948210 29457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:48:21.948347 29457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:48:21.948565 29457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:48:21.952791 29457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:48:21.953351 29457 leader_election.cc:302] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ada969f3d5f4076bd5233a91d976513; no voters:
I20260928 15:48:21.954026 29457 leader_election.cc:288] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:48:21.954164 29461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:48:21.955579 29461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 2 LEADER]: Becoming Leader. State: Replica: 0ada969f3d5f4076bd5233a91d976513, State: Running, Role: LEADER
I20260928 15:48:21.956275 29461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:48:21.956697 29457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:48:21.959142 29463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ada969f3d5f4076bd5233a91d976513. Latest consensus state: current_term: 2 leader_uuid: "0ada969f3d5f4076bd5233a91d976513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } } }
I20260928 15:48:21.959532 29463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: This master's current role is: LEADER
I20260928 15:48:21.959179 29462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0ada969f3d5f4076bd5233a91d976513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } } }
I20260928 15:48:21.959657 29462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: This master's current role is: LEADER
I20260928 15:48:21.960588 29468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:48:21.976581 29468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:48:21.978361 29468 catalog_manager.cc:1259] Loaded cluster ID: 1c360593c2d74f1fb237a906ced4cd74
I20260928 15:48:21.978658 29468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:48:21.983089 29468 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:48:21.984794 29468 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Loaded TSK: 0
I20260928 15:48:22.029327 29468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Generated new TSK 1
I20260928 15:48:22.029807 29468 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:48:22.295039 29459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:22.295382 29459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:22.295557 29459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:22.303380 29459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:22.303566 29459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:22.303748 29459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:48:22.315418 29459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:41567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=34239
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:22.317106 29459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:22.318873 29459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:22.335753 29484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:22.336148 29485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:22.336453 29487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:22.336905 29459 server_base.cc:1034] running on GCE node
I20260928 15:48:22.337641 29459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:22.338768 29459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:22.339987 29459 hybrid_clock.cc:648] HybridClock initialized: now 1790610502339929 us; error 45 us; skew 500 ppm
I20260928 15:48:22.340337 29459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:22 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610502, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:22.343812 29459 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:48:22.345593 29459 webserver.cc:460] Webserver started at http://127.13.5.65:34239/ using document root <none> and password file <none>
I20260928 15:48:22.346338 29459 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:22.346490 29459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:22.352559 29459 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:48:22.355581 29494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:22.357268 29459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:48:22.357455 29459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "eb10e06365564c23bf1b422f34a2c03f"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "221c8805f3cb13e50d57992e0786bbdc"
server_key_iv: "e2773694141c71798363db1b8da3f786"
server_key_version: "encryptionkey@0"
I20260928 15:48:22.358598 29459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:22.375998 29459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:22.380657 29459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:22.380952 29459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:22.382115 29459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:48:22.383533 29459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:48:22.383623 29459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:22.383773 29459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:48:22.383896 29459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:22.421026 29459 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:41567
I20260928 15:48:22.421121 29607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:41567 every 8 connection(s)
I20260928 15:48:22.422611 29459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:48:22.425570 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 29459
I20260928 15:48:22.426461 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:33071
--local_ip_for_outbound_sockets=127.13.5.66
--tserver_master_addrs=127.13.5.126:45959
--webserver_port=38825
--webserver_interface=127.13.5.66
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:48:22 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610502, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:48:22.452468 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:22.426096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:58301)
0928 15:48:22.428509 (+ 2413us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:22.428567 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:22.430492 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:22.432084 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:22.432097 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:22.432744 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:48:22.433332 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:22.433351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:22.435157 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:22.435162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:22.436847 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:22.436862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:22.438476 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:22.438487 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:22.438704 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:22.439513 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:48:22.439538 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:48:22.442150 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:48:22.447234 (+ 5084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:22.447243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:22.447257 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:22.448024 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:22.448707 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:22.448712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:22.448715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:22.448972 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:22.450049 (+ 1077us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:48:22.450056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:48:22.450406 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 15:48:22.450686 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 15:48:22.451170 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2090,"thread_start_us":1848,"threads_started":1}
I20260928 15:48:22.452937 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:22.425945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:58301 (local address 127.13.5.126:45959)
0928 15:48:22.426642 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:22.426664 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:22.429069 (+ 2405us) server_negotiation.cc:407] Connection header received
0928 15:48:22.431125 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:22.431138 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:22.431509 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:22.431928 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:22.433595 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:22.434951 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:22.437019 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:22.438159 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:22.438358 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:22.442490 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:22.442503 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:22.442527 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:22.442582 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:22.447042 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:22.448234 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:22.448248 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:22.448262 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:22.448532 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:22.449111 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:22.449118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:22.449123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:22.449880 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:22.450119 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:22.451005 (+ 886us) server_negotiation.cc:299] Negotiation successful
0928 15:48:22.451516 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
I20260928 15:48:22.455802 29608 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45959
I20260928 15:48:22.456178 29608 heartbeater.cc:464] Registering TS with master...
I20260928 15:48:22.457093 29608 heartbeater.cc:511] Master 127.13.5.126:45959 requested a full tablet report, sending...
I20260928 15:48:22.459656 29422 ts_manager.cc:194] Registered new tserver with Master: eb10e06365564c23bf1b422f34a2c03f (127.13.5.65:41567)
I20260928 15:48:22.461246 29422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:58301
W20260928 15:48:22.781118 29612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:22.781436 29612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:22.781539 29612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:22.788841 29612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:22.788952 29612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:22.789076 29612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:48:22.798974 29612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:33071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=38825
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:22.800273 29612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:22.801620 29612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:22.814935 29617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:22.814961 29618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:22.815018 29620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:22.815531 29612 server_base.cc:1034] running on GCE node
I20260928 15:48:22.816345 29612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:22.817386 29612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:22.818637 29612 hybrid_clock.cc:648] HybridClock initialized: now 1790610502818605 us; error 82 us; skew 500 ppm
I20260928 15:48:22.819000 29612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:22 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610502, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:22.822454 29612 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:48:22.824231 29612 webserver.cc:460] Webserver started at http://127.13.5.66:38825/ using document root <none> and password file <none>
I20260928 15:48:22.824975 29612 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:22.825140 29612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:22.830977 29612 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 15:48:22.833912 29627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:22.835397 29612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:48:22.835660 29612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "ad4bd7711d9a4b638edc27ccabd81875"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "15a809aab2d976b9e33ddbefc9aedbf6"
server_key_iv: "032f6bdcc28ad81fb8e404ea7a3ccd8f"
server_key_version: "encryptionkey@0"
I20260928 15:48:22.836835 29612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:22.856025 29612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:22.860912 29612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:22.861213 29612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:22.862860 29612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:48:22.864552 29612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:48:22.864682 29612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:22.864825 29612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:48:22.864924 29612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:22.902393 29612 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:33071
I20260928 15:48:22.902513 29739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:33071 every 8 connection(s)
I20260928 15:48:22.903946 29612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:48:22.906672 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 29612
I20260928 15:48:22.907401 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:34569
--local_ip_for_outbound_sockets=127.13.5.67
--tserver_master_addrs=127.13.5.126:45959
--webserver_port=35671
--webserver_interface=127.13.5.67
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:48:22 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610502, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:48:22.929188 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:22.906754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:45127 (local address 127.13.5.126:45959)
0928 15:48:22.906972 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:22.906992 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:22.908488 (+ 1496us) server_negotiation.cc:407] Connection header received
0928 15:48:22.911450 (+ 2962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:22.911454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:22.911554 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:22.911651 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:22.914013 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:22.914852 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:22.917206 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:22.917944 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:22.918100 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:22.922260 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:22.922265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:22.922268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:22.922305 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:22.925165 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:22.926253 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:22.926257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:22.926259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:22.926393 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:22.926955 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:22.926958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:22.926960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:22.927250 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:22.927365 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:22.928435 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0928 15:48:22.928776 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:48:22.930747 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:22.907338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:45127)
0928 15:48:22.908285 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:22.908340 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:22.911116 (+ 2776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:22.912009 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:22.912024 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:22.912843 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:48:22.913762 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:22.913791 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:22.915069 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:22.915077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:22.916930 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:22.916952 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:22.918200 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:22.918206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:22.918412 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:22.919159 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:48:22.919187 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:48:22.921949 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:48:22.925335 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:22.925348 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:22.925370 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:22.926093 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:22.926512 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:22.926520 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:22.926527 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:22.926807 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:22.927389 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:48:22.927401 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:48:22.927954 (+ 553us) client_negotiation.cc:769] Sending connection context
0928 15:48:22.928305 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 15:48:22.928991 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":401,"threads_started":1}
I20260928 15:48:22.932497 29740 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45959
I20260928 15:48:22.932850 29740 heartbeater.cc:464] Registering TS with master...
I20260928 15:48:22.933648 29740 heartbeater.cc:511] Master 127.13.5.126:45959 requested a full tablet report, sending...
I20260928 15:48:22.935196 29422 ts_manager.cc:194] Registered new tserver with Master: ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
I20260928 15:48:22.936103 29422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:45127
W20260928 15:48:23.264154 29743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:23.264434 29743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:23.264532 29743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:23.271816 29743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:23.271988 29743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:23.272109 29743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:48:23.282146 29743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:34569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=35671
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:23.283442 29743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:23.284770 29743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:23.299042 29748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:23.299343 29749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:23.299674 29751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:23.300546 29743 server_base.cc:1034] running on GCE node
I20260928 15:48:23.301203 29743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:23.302372 29743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:23.303625 29743 hybrid_clock.cc:648] HybridClock initialized: now 1790610503303579 us; error 38 us; skew 500 ppm
I20260928 15:48:23.304039 29743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:23 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610503, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:23.307410 29743 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:48:23.309083 29743 webserver.cc:460] Webserver started at http://127.13.5.67:35671/ using document root <none> and password file <none>
I20260928 15:48:23.309850 29743 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:23.310024 29743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:23.315785 29743 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:48:23.318753 29758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:23.320564 29743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:48:23.320765 29743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "9e2729c54e3b43ceaa01a9c6926d55b9"
format_stamp: "Formatted at 2026-09-28 15:48:00 on dist-test-slave-j968"
server_key: "ebb703d9ab5f385b5d54fa5e754e1920"
server_key_iv: "a77c84c5e24e0920657c04468b48e7ac"
server_key_version: "encryptionkey@0"
I20260928 15:48:23.321847 29743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:23.340728 29743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:23.345415 29743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:23.345656 29743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:23.346601 29743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:48:23.348069 29743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:48:23.348258 29743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:23.348366 29743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:48:23.348465 29743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:23.382972 29743 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:34569
I20260928 15:48:23.383059 29872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:34569 every 8 connection(s)
I20260928 15:48:23.384438 29743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:48:23.388190 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 29743
Sep 28 15:48:23 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610503, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:48:23.409480 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.387188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:46673 (local address 127.13.5.126:45959)
0928 15:48:23.387375 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.387380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.388682 (+ 1302us) server_negotiation.cc:407] Connection header received
0928 15:48:23.390842 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.390847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.390943 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.391057 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:23.392731 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.393938 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.396258 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.396982 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.397125 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.401125 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:23.401135 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:23.401145 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:23.401198 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:23.405214 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.406296 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.406300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.406302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.406421 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.406898 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.406903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.406906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.407314 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:23.407452 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.408757 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.408997 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:48:23.409899 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.389297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.13.5.126:45959)
0928 15:48:23.389789 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.389796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.392517 (+ 2721us) server_negotiation.cc:407] Connection header received
0928 15:48:23.392662 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.392668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.392748 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.392830 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:23.394820 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.396001 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.398226 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.398955 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.399104 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.403111 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:23.403116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:23.403120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:23.403166 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:23.406399 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.407230 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.407234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.407236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.407347 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.407793 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.407797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.407800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.408172 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:23.408429 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.408887 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.409123 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":273,"threads_started":1}
I20260928 15:48:23.410331 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.387554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:46673)
0928 15:48:23.388482 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.388544 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:23.390562 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:23.391297 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:23.391306 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:23.391802 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:48:23.392525 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.392549 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.394149 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.394153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.396074 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.396089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.397205 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.397210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.397417 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.398209 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:48:23.398241 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:48:23.400839 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:48:23.405389 (+ 4550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:23.405403 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:23.405428 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:23.406149 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:23.406570 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:23.406573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:23.406576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:23.406770 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:23.407512 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:48:23.407521 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:48:23.407978 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 15:48:23.408290 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 15:48:23.408827 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":362,"threads_started":1}
I20260928 15:48:23.411978 29874 heartbeater.cc:347] Connected to a master server at 127.13.5.126:45959
I20260928 15:48:23.412318 29874 heartbeater.cc:464] Registering TS with master...
I20260928 15:48:23.413144 29874 heartbeater.cc:511] Master 127.13.5.126:45959 requested a full tablet report, sending...
I20260928 15:48:23.414542 29422 ts_manager.cc:194] Registered new tserver with Master: 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569)
I20260928 15:48:23.415373 29422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:46673
I20260928 15:48:23.425570 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:48:23.432679 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.426711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.13.5.126:45959)
0928 15:48:23.426945 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.426951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.427161 (+ 210us) server_negotiation.cc:407] Connection header received
0928 15:48:23.427341 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.427344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.427519 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.427647 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:48:23.428189 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.428939 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.430549 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.430969 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.431067 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.431584 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:48:23.431928 (+ 344us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 15:48:23.432330 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49274: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 15:48:23.432829 29877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49274: token expired
W20260928 15:48:23.433444 29343 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.5.126:45959)
I20260928 15:48:23.447134 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.433920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.13.5.126:45959)
0928 15:48:23.434089 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.434093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.434114 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:48:23.434270 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.434274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.434335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.434417 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:23.435988 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.436820 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.438222 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.438716 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.438817 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.441719 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:23.441726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:23.441730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:23.441771 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:23.444478 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.445198 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.445202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.445204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.445330 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.445738 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.445741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.445743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.446087 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:23.446276 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.446608 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.446785 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:48:23.451098 29422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:48:23.458652 29422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:48:23.464803 29608 heartbeater.cc:503] Master 127.13.5.126:45959 was elected leader, sending a full tablet report...
I20260928 15:48:23.492265 29887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.471642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.13.5.65:41567)
0928 15:48:23.484800 (+ 13158us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.484806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.484854 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:48:23.484958 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.484967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.485188 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.485398 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:23.486362 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.487535 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.489637 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.490874 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.491042 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.491297 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.491575 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.491760 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12979,"thread_start_us":13929,"threads_started":1}
I20260928 15:48:23.492985 29888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.472438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.13.5.66:33071)
0928 15:48:23.483659 (+ 11221us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.483669 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.483696 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:48:23.483800 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.483809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.484085 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.484297 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:23.484919 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.486477 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.489951 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.491419 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.491544 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.491737 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.492464 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.492630 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11039,"thread_start_us":10906,"threads_started":1}
I20260928 15:48:23.492987 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.471285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.13.5.67:34569)
0928 15:48:23.472434 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.472443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.482978 (+ 10535us) server_negotiation.cc:407] Connection header received
0928 15:48:23.483117 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.483127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.483468 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.483731 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:23.484734 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.486560 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.489950 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.491416 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.491544 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.491796 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.492476 (+ 680us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.492631 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":506,"threads_started":1}
I20260928 15:48:23.493115 29885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.470985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58596)
0928 15:48:23.480566 (+ 9581us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.480595 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:23.480716 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:23.485833 (+ 5117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:23.485838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:23.485864 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:23.486186 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.486196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.487683 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.487689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.489434 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.489450 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.491099 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.491105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.491308 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.491378 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 15:48:23.491530 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 15:48:23.492772 (+ 1242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9413,"mutex_wait_us":7,"thread_start_us":1692,"threads_started":1}
I20260928 15:48:23.493714 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.473015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33071 (local address 127.0.0.1:55620)
0928 15:48:23.483027 (+ 10012us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.483045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:23.483149 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:23.484280 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:23.484284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:23.484316 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:23.484738 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.484749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.486619 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.486626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.489753 (+ 3127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.489768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.491934 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.491943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.492154 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.492261 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 15:48:23.492348 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:48:23.493293 (+ 945us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9874,"mutex_wait_us":1123,"thread_start_us":9718,"threads_started":1}
I20260928 15:48:23.494076 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.470976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:34569 (local address 127.0.0.1:59818)
0928 15:48:23.472840 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.472884 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:23.479892 (+ 7008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:23.483800 (+ 3908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:23.483810 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:23.483860 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:23.484477 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.484497 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.486760 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.486766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.489752 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.489768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.491974 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.491979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.492147 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.492262 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 15:48:23.492341 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 15:48:23.493579 (+ 1238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"mutex_wait_us":1285,"thread_start_us":11379,"threads_started":1}
I20260928 15:48:23.497552 29808 tablet_service.cc:1467] Processing CreateTablet for tablet d86871864b1d4250a5545b8c29c85ff5 (DEFAULT_TABLE table=test-table [id=29c8827e1acb445c975899808f36ecb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:48:23.497723 29542 tablet_service.cc:1467] Processing CreateTablet for tablet d86871864b1d4250a5545b8c29c85ff5 (DEFAULT_TABLE table=test-table [id=29c8827e1acb445c975899808f36ecb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:48:23.498075 29675 tablet_service.cc:1467] Processing CreateTablet for tablet d86871864b1d4250a5545b8c29c85ff5 (DEFAULT_TABLE table=test-table [id=29c8827e1acb445c975899808f36ecb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:48:23.498934 29808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86871864b1d4250a5545b8c29c85ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:48:23.499188 29542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86871864b1d4250a5545b8c29c85ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:48:23.499248 29675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d86871864b1d4250a5545b8c29c85ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:48:23.511905 29890 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Bootstrap starting.
I20260928 15:48:23.514667 29891 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Bootstrap starting.
I20260928 15:48:23.516942 29890 tablet_bootstrap.cc:654] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Neither blocks nor log segments found. Creating new log.
I20260928 15:48:23.518285 29890 log.cc:824] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:23.518337 29891 tablet_bootstrap.cc:654] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Neither blocks nor log segments found. Creating new log.
I20260928 15:48:23.519189 29892 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Bootstrap starting.
I20260928 15:48:23.519814 29891 log.cc:824] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:23.522859 29892 tablet_bootstrap.cc:654] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Neither blocks nor log segments found. Creating new log.
I20260928 15:48:23.522987 29890 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: No bootstrap required, opened a new log
I20260928 15:48:23.523131 29891 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: No bootstrap required, opened a new log
I20260928 15:48:23.523309 29890 ts_tablet_manager.cc:1397] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260928 15:48:23.523423 29891 ts_tablet_manager.cc:1397] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 15:48:23.524132 29892 log.cc:824] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:23.527395 29892 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: No bootstrap required, opened a new log
I20260928 15:48:23.527693 29892 ts_tablet_manager.cc:1397] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 15:48:23.531598 29890 raft_consensus.cc:348] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.532044 29890 raft_consensus.cc:374] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:48:23.532186 29890 raft_consensus.cc:729] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2729c54e3b43ceaa01a9c6926d55b9, State: Initialized, Role: FOLLOWER
I20260928 15:48:23.532845 29890 consensus_queue.cc:260] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.534098 29891 raft_consensus.cc:348] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.534575 29891 raft_consensus.cc:374] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:48:23.534689 29891 raft_consensus.cc:729] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb10e06365564c23bf1b422f34a2c03f, State: Initialized, Role: FOLLOWER
I20260928 15:48:23.534713 29874 heartbeater.cc:503] Master 127.13.5.126:45959 was elected leader, sending a full tablet report...
I20260928 15:48:23.535408 29890 ts_tablet_manager.cc:1428] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Time spent starting tablet: real 0.012s user 0.005s sys 0.008s
I20260928 15:48:23.535394 29891 consensus_queue.cc:260] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.537119 29891 ts_tablet_manager.cc:1428] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260928 15:48:23.538300 29892 raft_consensus.cc:348] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.538803 29892 raft_consensus.cc:374] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:48:23.538915 29892 raft_consensus.cc:729] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4bd7711d9a4b638edc27ccabd81875, State: Initialized, Role: FOLLOWER
I20260928 15:48:23.539508 29892 consensus_queue.cc:260] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.540876 29740 heartbeater.cc:503] Master 127.13.5.126:45959 was elected leader, sending a full tablet report...
I20260928 15:48:23.541500 29892 ts_tablet_manager.cc:1428] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260928 15:48:23.620050 29896 raft_consensus.cc:482] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:48:23.620383 29896 raft_consensus.cc:504] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.622475 29896 leader_election.cc:288] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb10e06365564c23bf1b422f34a2c03f (127.13.5.65:41567), ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
I20260928 15:48:23.629823 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.622846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.13.5.67:54069)
0928 15:48:23.623286 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.623310 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:23.623506 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:23.624000 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:23.624004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:23.624028 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:23.624353 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.624369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.626077 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.626082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.627791 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.627803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.629013 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.629023 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.629249 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.629275 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:48:23.629349 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 15:48:23.629499 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":59}
I20260928 15:48:23.629817 29888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.623157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:50649 (local address 127.13.5.66:33071)
0928 15:48:23.623475 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.623483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.624140 (+ 657us) server_negotiation.cc:407] Connection header received
0928 15:48:23.624344 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.624350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.624433 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.624529 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:23.625277 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.626179 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.628092 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.628878 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.629030 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.629124 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.629438 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.629564 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260928 15:48:23.629817 29887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.623062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:54069 (local address 127.13.5.65:41567)
0928 15:48:23.623290 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.623305 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.623406 (+ 101us) server_negotiation.cc:407] Connection header received
0928 15:48:23.623715 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.623721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.623817 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.623978 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:48:23.624549 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.625700 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.627968 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.628781 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.628891 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.629000 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.629405 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.629549 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:48:23.629908 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.623071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33071 (local address 127.13.5.67:50649)
0928 15:48:23.624017 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.624036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:23.624166 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:23.624551 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:23.624555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:23.624581 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:48:23.625079 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.625101 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.626339 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.626347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.627909 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:23.627926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.629098 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.629102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:23.629265 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.629289 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:48:23.629391 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:48:23.629519 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":70,"thread_start_us":492,"threads_started":1}
I20260928 15:48:23.631150 29695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86871864b1d4250a5545b8c29c85ff5" candidate_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4bd7711d9a4b638edc27ccabd81875" is_pre_election: true
I20260928 15:48:23.631150 29563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86871864b1d4250a5545b8c29c85ff5" candidate_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb10e06365564c23bf1b422f34a2c03f" is_pre_election: true
I20260928 15:48:23.631650 29695 raft_consensus.cc:2454] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2729c54e3b43ceaa01a9c6926d55b9 in term 0.
I20260928 15:48:23.631691 29563 raft_consensus.cc:2454] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2729c54e3b43ceaa01a9c6926d55b9 in term 0.
I20260928 15:48:23.632344 29761 leader_election.cc:302] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e2729c54e3b43ceaa01a9c6926d55b9, ad4bd7711d9a4b638edc27ccabd81875; no voters:
I20260928 15:48:23.632756 29896 raft_consensus.cc:2781] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:48:23.632905 29896 raft_consensus.cc:482] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:48:23.633036 29896 raft_consensus.cc:3037] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:48:23.635252 29896 raft_consensus.cc:504] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.636025 29896 leader_election.cc:288] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [CANDIDATE]: Term 1 election: Requested vote from peers eb10e06365564c23bf1b422f34a2c03f (127.13.5.65:41567), ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
W20260928 15:48:23.636188 29875 tablet.cc:2369] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:48:23.636401 29563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86871864b1d4250a5545b8c29c85ff5" candidate_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb10e06365564c23bf1b422f34a2c03f"
I20260928 15:48:23.636627 29563 raft_consensus.cc:3037] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:48:23.636772 29695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86871864b1d4250a5545b8c29c85ff5" candidate_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4bd7711d9a4b638edc27ccabd81875"
I20260928 15:48:23.636991 29695 raft_consensus.cc:3037] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:48:23.639330 29563 raft_consensus.cc:2454] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2729c54e3b43ceaa01a9c6926d55b9 in term 1.
I20260928 15:48:23.639650 29695 raft_consensus.cc:2454] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2729c54e3b43ceaa01a9c6926d55b9 in term 1.
I20260928 15:48:23.639725 29760 leader_election.cc:302] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e2729c54e3b43ceaa01a9c6926d55b9, eb10e06365564c23bf1b422f34a2c03f; no voters:
I20260928 15:48:23.640105 29896 raft_consensus.cc:2781] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:48:23.640535 29896 raft_consensus.cc:686] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2729c54e3b43ceaa01a9c6926d55b9, State: Running, Role: LEADER
I20260928 15:48:23.641007 29896 consensus_queue.cc:237] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:23.645695 29422 catalog_manager.cc:5495] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } health_report { overall_health: UNKNOWN } } }
W20260928 15:48:23.656119 29741 tablet.cc:2369] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:48:23.675231 29609 tablet.cc:2369] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:48:23.687134 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.681897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.13.5.67:34569)
0928 15:48:23.682100 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.682105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.682123 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:48:23.682286 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.682290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.682376 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.682515 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:48:23.682910 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.683655 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.685087 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.685473 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.685571 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.686142 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:48:23.686633 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:48:23.686748 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.686840 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.686908 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:48:23.695930 29695 raft_consensus.cc:1261] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 1 FOLLOWER]: Refusing update from remote peer 9e2729c54e3b43ceaa01a9c6926d55b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:48:23.695930 29563 raft_consensus.cc:1261] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Refusing update from remote peer 9e2729c54e3b43ceaa01a9c6926d55b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:48:23.697117 29900 consensus_queue.cc:1035] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:48:23.697515 29896 consensus_queue.cc:1035] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:48:23.717994 29903 mvcc.cc:205] Tried to move back new op lower bound from 7334340623126618112 to 7334340622918168576. Current Snapshot: MvccSnapshot[applied={T|T < 7334340623126618112}]
I20260928 15:48:23.729502 29421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:48:23.729820 29421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:48:23.733942 29421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Sending DeleteTablet for 3 replicas of tablet d86871864b1d4250a5545b8c29c85ff5
I20260928 15:48:23.735189 29808 tablet_service.cc:1514] Processing DeleteTablet for tablet d86871864b1d4250a5545b8c29c85ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:48:23 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:59818
I20260928 15:48:23.735327 29542 tablet_service.cc:1514] Processing DeleteTablet for tablet d86871864b1d4250a5545b8c29c85ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:48:23 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:58596
I20260928 15:48:23.735327 29675 tablet_service.cc:1514] Processing DeleteTablet for tablet d86871864b1d4250a5545b8c29c85ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:48:23 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:55620
I20260928 15:48:23.736910 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 29459
I20260928 15:48:23.752676 29914 tablet_replica.cc:330] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: stopping tablet replica
I20260928 15:48:23.753393 29914 raft_consensus.cc:2229] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:48:23.754143 29914 raft_consensus.cc:2258] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:48:23.754837 29912 tablet_replica.cc:330] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: stopping tablet replica
I20260928 15:48:23.755712 29912 raft_consensus.cc:2229] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:48:23.758282 29912 raft_consensus.cc:2258] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:48:23.759367 29914 ts_tablet_manager.cc:1905] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:48:23.759936 29887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.738179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.13.5.65:41567)
0928 15:48:23.738384 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.738393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.738433 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:48:23.738508 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.738516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.738618 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.738720 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:23.740914 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.742590 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.745082 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.746046 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.746172 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.750577 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:23.750603 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:23.750632 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:23.750695 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:23.755180 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.756413 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.756424 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.756427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.756610 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.757118 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.757123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.757126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.757617 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:23.757923 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.759191 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.759495 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:48:23.760975 29603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:48:23.761569 29912 ts_tablet_manager.cc:1905] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:48:23.768715 29914 ts_tablet_manager.cc:1918] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:48:23.768962 29914 log.cc:1196] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/d86871864b1d4250a5545b8c29c85ff5
I20260928 15:48:23.769586 29914 ts_tablet_manager.cc:1939] T d86871864b1d4250a5545b8c29c85ff5 P ad4bd7711d9a4b638edc27ccabd81875: Deleting consensus metadata
I20260928 15:48:23.771799 29407 catalog_manager.cc:4841] TS ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071): tablet d86871864b1d4250a5545b8c29c85ff5 (table test-table [id=29c8827e1acb445c975899808f36ecb2]) successfully deleted
I20260928 15:48:23.772584 29912 ts_tablet_manager.cc:1918] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:48:23.772923 29912 log.cc:1196] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/d86871864b1d4250a5545b8c29c85ff5
I20260928 15:48:23.773685 29912 ts_tablet_manager.cc:1939] T d86871864b1d4250a5545b8c29c85ff5 P 9e2729c54e3b43ceaa01a9c6926d55b9: Deleting consensus metadata
I20260928 15:48:23.776118 29405 catalog_manager.cc:4841] TS 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569): tablet d86871864b1d4250a5545b8c29c85ff5 (table test-table [id=29c8827e1acb445c975899808f36ecb2]) successfully deleted
I20260928 15:48:23.868201 29913 tablet_replica.cc:330] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: stopping tablet replica
I20260928 15:48:23.868577 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 29459
I20260928 15:48:23.888131 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 29612
W20260928 15:48:23.888130 29406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.5.65:41567 (error 108)
I20260928 15:48:23.903192 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.902712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58620)
0928 15:48:23.902879 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.903012 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:48:23.907106 29406 catalog_manager.cc:4584] TS eb10e06365564c23bf1b422f34a2c03f (127.13.5.65:41567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d86871864b1d4250a5545b8c29c85ff5: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
I20260928 15:48:23.912599 29888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.889113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.13.5.66:33071)
0928 15:48:23.889303 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:23.889309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:23.889388 (+ 79us) server_negotiation.cc:407] Connection header received
0928 15:48:23.889613 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:23.889623 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:23.889733 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:23.889866 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:23.892466 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.893880 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.896538 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:23.897384 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:23.897522 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:23.901998 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:23.902016 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:23.902036 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:23.902099 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:23.907881 (+ 5782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.908866 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.908875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.908879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.909035 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:23.909451 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:23.909456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:23.909458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:23.910320 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:23.910623 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:23.911975 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0928 15:48:23.912201 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:48:23.913820 29735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:48:23.961135 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.960607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58634)
0928 15:48:23.960845 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.960957 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 15:48:23.962364 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:23.961912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58640)
0928 15:48:23.962090 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:23.962203 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 15:48:24.008649 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 29612
I20260928 15:48:24.023137 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:24.022528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58656)
0928 15:48:24.022797 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:24.022928 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 15:48:24.024405 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:24.023984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58666)
0928 15:48:24.024155 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:24.024248 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 15:48:24.028707 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 29743
I20260928 15:48:24.046306 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:24.029537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.13.5.67:34569)
0928 15:48:24.029740 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:24.029746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:24.029846 (+ 100us) server_negotiation.cc:407] Connection header received
0928 15:48:24.030006 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:24.030012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:24.030107 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:24.030223 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:24.031802 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:24.033067 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:24.034614 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:24.035328 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:24.035494 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:24.038309 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:24.038325 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:24.038348 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:24.038407 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:24.042866 (+ 4459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:24.043608 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:24.043617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:24.043621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:24.043815 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:24.044536 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:24.044541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:24.044545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:24.045027 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:24.045354 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:24.045606 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:48:24.045851 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:48:24.047150 29868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:48:24.105253 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:24.104674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58672)
0928 15:48:24.104955 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:24.105059 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 15:48:24.106421 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:24.106016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58680)
0928 15:48:24.106219 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:24.106301 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 15:48:24.183313 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 29743
I20260928 15:48:24.204257 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 29388
I20260928 15:48:24.205523 29452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:48:24.322741 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:24.321964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:41567 (local address 127.0.0.1:58696)
0928 15:48:24.322345 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:24.322479 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260928 15:48:24.323246 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 29388
I20260928 15:48:44.348219 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45959
--webserver_interface=127.13.5.126
--webserver_port=38459
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:48:44.692577 29920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:44.692906 29920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:44.693076 29920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:44.700609 29920 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:48:44.700711 29920 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:48:44.700799 29920 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:48:44.700881 29920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:48:44.700985 29920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:44.701107 29920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:44.701205 29920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:48:44.701300 29920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:48:44.711493 29920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=38459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:44.712935 29920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:44.715049 29920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:44.729099 29926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:44.729141 29925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:44.729418 29928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:44.730044 29920 server_base.cc:1034] running on GCE node
I20260928 15:48:44.730752 29920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:44.732261 29920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:44.733536 29920 hybrid_clock.cc:648] HybridClock initialized: now 1790610524733444 us; error 39 us; skew 500 ppm
I20260928 15:48:44.733873 29920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:44 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610524, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:44.737356 29920 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:48:44.738854 29920 webserver.cc:460] Webserver started at http://127.13.5.126:38459/ using document root <none> and password file <none>
I20260928 15:48:44.739565 29920 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:44.739657 29920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:44.745391 29920 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:48:44.748306 29935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:44.749794 29920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:48:44.749997 29920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "0ada969f3d5f4076bd5233a91d976513"
format_stamp: "Formatted at 2026-09-28 15:47:58 on dist-test-slave-j968"
server_key: "38859c7e2a1674a0a4040008b0d06fb8"
server_key_iv: "b9f32f805f2315a3185ab0fb35d00c89"
server_key_version: "encryptionkey@0"
I20260928 15:48:44.750990 29920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:44.790629 29920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:44.795187 29920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:44.795482 29920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:44.815260 29920 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:45959
I20260928 15:48:44.815356 29987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:45959 every 8 connection(s)
I20260928 15:48:44.816895 29920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:48:44.824189 29988 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:48:44.826701 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 29920
I20260928 15:48:44.827796 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:41567
--local_ip_for_outbound_sockets=127.13.5.65
--tserver_master_addrs=127.13.5.126:45959
--webserver_port=34239
--webserver_interface=127.13.5.65
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:48:44.828428 29988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Bootstrap starting.
I20260928 15:48:44.852211 29988 log.cc:824] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:44.869623 29988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:48:44.870196 29988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Bootstrap complete.
I20260928 15:48:44.879676 29988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:48:44.880692 29988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ada969f3d5f4076bd5233a91d976513, State: Initialized, Role: FOLLOWER
I20260928 15:48:44.881423 29988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:48:44.881681 29988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:48:44.881816 29988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:48:44.882015 29988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:48:44.886053 29988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:48:44.886579 29988 leader_election.cc:302] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ada969f3d5f4076bd5233a91d976513; no voters:
I20260928 15:48:44.887320 29988 leader_election.cc:288] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:48:44.887480 29992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:48:44.888875 29992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [term 3 LEADER]: Becoming Leader. State: Replica: 0ada969f3d5f4076bd5233a91d976513, State: Running, Role: LEADER
I20260928 15:48:44.889689 29988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:48:44.889542 29992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } }
I20260928 15:48:44.892352 29993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0ada969f3d5f4076bd5233a91d976513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } } }
I20260928 15:48:44.892786 29993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: This master's current role is: LEADER
I20260928 15:48:44.897409 29994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ada969f3d5f4076bd5233a91d976513. Latest consensus state: current_term: 3 leader_uuid: "0ada969f3d5f4076bd5233a91d976513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ada969f3d5f4076bd5233a91d976513" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 45959 } } }
I20260928 15:48:44.897778 29994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513 [sys.catalog]: This master's current role is: LEADER
I20260928 15:48:44.900372 29998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:48:44.909507 29998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:48:44.911199 29998 catalog_manager.cc:1259] Loaded cluster ID: 1c360593c2d74f1fb237a906ced4cd74
I20260928 15:48:44.911372 29998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:48:44.915895 29998 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:48:44.917948 29998 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Loaded TSK: 0
I20260928 15:48:44.918179 29998 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ada969f3d5f4076bd5233a91d976513: Loaded TSK: 1
F20260928 15:48:44.919580 29998 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1b398da9ad google::LogMessage::Fail() at ??:0
@ 0x7f1b398dd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1b398da4de google::LogMessage::Flush() at ??:0
@ 0x7f1b398dba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1b3d63e7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1b3d62d73e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260928 15:48:44.940303 30010 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f1b4f3091c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1b4f308783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1b4f3942a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f1b4f3940c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1b4f3de5e9 std::function<>::operator()() at ??:0
@ 0x7f1b4f30ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1b4f30ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1b4f392610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1b4f392439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1b3c526345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1b3c52991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1b3c529799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1b3c504eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1b3f7de6db start_thread at ??:0
@ 0x7f1b3877461f clone at ??:0
W20260928 15:48:45.225401 29990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:45.225739 29990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:45.225924 29990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:45.233655 29990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:45.233829 29990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:45.234019 29990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:48:45.244203 29990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:41567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=34239
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:45.245651 29990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:45.247027 29990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:45.260689 30038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:45.260762 30035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:45.260726 30036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:45.261523 29990 server_base.cc:1034] running on GCE node
I20260928 15:48:45.262202 29990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:45.263249 29990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:45.264480 29990 hybrid_clock.cc:648] HybridClock initialized: now 1790610525264415 us; error 76 us; skew 500 ppm
I20260928 15:48:45.265027 29990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:45 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610525, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:45.268249 29990 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:48:45.269658 29990 webserver.cc:460] Webserver started at http://127.13.5.65:34239/ using document root <none> and password file <none>
I20260928 15:48:45.270319 29990 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:45.270431 29990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:45.276134 29990 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:48:45.278908 30045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:45.280339 29990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:48:45.280534 29990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "eb10e06365564c23bf1b422f34a2c03f"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "221c8805f3cb13e50d57992e0786bbdc"
server_key_iv: "e2773694141c71798363db1b8da3f786"
server_key_version: "encryptionkey@0"
I20260928 15:48:45.281504 29990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:45.317247 29990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:45.321563 29990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:45.321861 29990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:45.322976 29990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:48:45.325032 30052 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 15:48:45.331152 29990 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 15:48:45.331284 29990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260928 15:48:45.331463 29990 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 15:48:45.334913 29990 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 15:48:45.335003 29990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.000s
I20260928 15:48:45.335444 30052 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Bootstrap starting.
I20260928 15:48:45.363502 30052 log.cc:824] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:48:45.378654 30161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:41567 every 8 connection(s)
I20260928 15:48:45.378644 29990 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:41567
I20260928 15:48:45.378647 30052 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:48:45.379395 30052 tablet_bootstrap.cc:492] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Bootstrap complete.
I20260928 15:48:45.380167 29990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:48:45.380898 30052 ts_tablet_manager.cc:1397] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Time spent bootstrapping tablet: real 0.046s user 0.034s sys 0.009s
I20260928 15:48:45.386174 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:45.383596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:46069)
0928 15:48:45.384570 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:45.384913 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":433,"threads_started":1}
W20260928 15:48:45.386900 30162 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:45959 (0 consecutive failures): Network error: Failed to ping master at 127.13.5.126:45959: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
I20260928 15:48:45.387766 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 29990
I20260928 15:48:45.388033 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:45.387554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:37431)
0928 15:48:45.387747 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:45.387844 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 15:48:45.388669 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:33071
--local_ip_for_outbound_sockets=127.13.5.66
--tserver_master_addrs=127.13.5.126:45959
--webserver_port=38825
--webserver_interface=127.13.5.66
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:48:45.389236 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:45.388688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:49457)
0928 15:48:45.388882 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:45.389031 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 15:48:45.389573 30162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:48:45.390221 30052 raft_consensus.cc:348] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:45.391147 30052 raft_consensus.cc:729] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb10e06365564c23bf1b422f34a2c03f, State: Initialized, Role: FOLLOWER
I20260928 15:48:45.391757 30052 consensus_queue.cc:260] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:45.393513 30052 ts_tablet_manager.cc:1428] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 15:48:45.393601 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:45.393122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:33409)
0928 15:48:45.393317 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:45.393413 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:48:45.394580 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:45.394171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:39907)
0928 15:48:45.394336 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:45.394424 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:48:45.727809 30166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:45.728152 30166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:45.728253 30166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:45.735667 30166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:45.735797 30166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:45.736013 30166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:48:45.746503 30166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:33071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=38825
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:45.747845 30166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:45.749192 30166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:45.763173 30172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:45.763181 30174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:45.763633 30171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:45.764293 30166 server_base.cc:1034] running on GCE node
I20260928 15:48:45.765040 30166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:45.766072 30166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:45.767351 30166 hybrid_clock.cc:648] HybridClock initialized: now 1790610525767298 us; error 40 us; skew 500 ppm
I20260928 15:48:45.767750 30166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:45 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610525, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:45.771206 30166 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:48:45.773003 30166 webserver.cc:460] Webserver started at http://127.13.5.66:38825/ using document root <none> and password file <none>
I20260928 15:48:45.773792 30166 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:45.773952 30166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:45.779983 30166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:48:45.783046 30181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:45.784549 30166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:48:45.784781 30166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "ad4bd7711d9a4b638edc27ccabd81875"
format_stamp: "Formatted at 2026-09-28 15:47:59 on dist-test-slave-j968"
server_key: "15a809aab2d976b9e33ddbefc9aedbf6"
server_key_iv: "032f6bdcc28ad81fb8e404ea7a3ccd8f"
server_key_version: "encryptionkey@0"
I20260928 15:48:45.785879 30166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:45.808349 30166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:45.812827 30166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:45.813129 30166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:45.814275 30166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:48:45.815742 30166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:48:45.815836 30166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:45.815934 30166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:48:45.816037 30166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:45.852010 30166 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:33071
I20260928 15:48:45.852135 30294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:33071 every 8 connection(s)
I20260928 15:48:45.853586 30166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:48:45.855630 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 30166
I20260928 15:48:45.856429 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:34569
--local_ip_for_outbound_sockets=127.13.5.67
--tserver_master_addrs=127.13.5.126:45959
--webserver_port=35671
--webserver_interface=127.13.5.67
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:33243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:48:45.859539 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:45.857342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:50745)
0928 15:48:45.858219 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:45.858518 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":359,"threads_started":1}
W20260928 15:48:45.860263 30295 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:45959 (0 consecutive failures): Network error: Failed to ping master at 127.13.5.126:45959: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
I20260928 15:48:45.861217 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:45.860869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:37043)
0928 15:48:45.861028 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:45.861105 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 15:48:45.862162 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:45.861813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:40933)
0928 15:48:45.861963 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:45.862049 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:48:45.862425 30295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:48:46.201865 30298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:48:46.202194 30298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:48:46.202320 30298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:48:46.209672 30298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:48:46.209789 30298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:48:46.209903 30298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:48:46.219884 30298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:33243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:34569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=35671
--tserver_master_addrs=127.13.5.126:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:48:46.221161 30298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:48:46.222503 30298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:48:46.236284 30306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:46.236325 30304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:48:46.236495 30303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:48:46.236968 30298 server_base.cc:1034] running on GCE node
I20260928 15:48:46.237617 30298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:48:46.238601 30298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:48:46.239841 30298 hybrid_clock.cc:648] HybridClock initialized: now 1790610526239792 us; error 64 us; skew 500 ppm
I20260928 15:48:46.240224 30298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:48:46 dist-test-slave-j968 krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610526, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:48:46.243536 30298 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:48:46.245126 30298 webserver.cc:460] Webserver started at http://127.13.5.67:35671/ using document root <none> and password file <none>
I20260928 15:48:46.245800 30298 fs_manager.cc:359] Metadata directory not provided
I20260928 15:48:46.245900 30298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:48:46.251466 30298 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 15:48:46.254470 30313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:46.256054 30298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:48:46.256228 30298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "9e2729c54e3b43ceaa01a9c6926d55b9"
format_stamp: "Formatted at 2026-09-28 15:48:00 on dist-test-slave-j968"
server_key: "ebb703d9ab5f385b5d54fa5e754e1920"
server_key_iv: "a77c84c5e24e0920657c04468b48e7ac"
server_key_version: "encryptionkey@0"
I20260928 15:48:46.257336 30298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:48:46.294474 30298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:48:46.299053 30298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:48:46.299347 30298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:48:46.300443 30298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:48:46.301851 30298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:48:46.301971 30298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:46.302107 30298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:48:46.302202 30298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:48:46.336697 30298 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:34569
I20260928 15:48:46.336796 30427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:34569 every 8 connection(s)
I20260928 15:48:46.338297 30298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:48:46.343602 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.341541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:48569)
0928 15:48:46.342357 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:46.342659 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":320,"threads_started":1}
I20260928 15:48:46.343667 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 30298
W20260928 15:48:46.344476 30428 heartbeater.cc:644] Failed to heartbeat to 127.13.5.126:45959 (0 consecutive failures): Network error: Failed to ping master at 127.13.5.126:45959: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
I20260928 15:48:46.345468 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.345057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:47557)
0928 15:48:46.345237 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:46.345328 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 15:48:46.346293 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.345963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:59217)
0928 15:48:46.346097 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:46.346175 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:48:46.346527 30428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:48:46.396379 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.395477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:53413)
0928 15:48:46.396076 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:46.396171 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":276,"threads_started":1}
I20260928 15:48:46.827101 30433 raft_consensus.cc:482] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:48:46.827469 30433 raft_consensus.cc:504] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:46.829391 30433 leader_election.cc:288] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569), ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
Sep 28 15:48:46 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610525, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
Sep 28 15:48:46 dist-test-slave-j968 krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610525, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:48:46.854092 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.829848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33071 (local address 127.13.5.65:34731)
0928 15:48:46.831809 (+ 1961us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:46.831830 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:46.832000 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:46.835324 (+ 3324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:46.835334 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:46.836202 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:48:46.836955 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:46.836978 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:46.838651 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:46.838658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:46.840411 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:46.840429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:46.841450 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:46.841457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:46.841721 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:46.842421 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:48:46.842444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:48:46.845056 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:48:46.849650 (+ 4594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:46.849660 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:46.849677 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:46.850642 (+ 965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:46.851253 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:46.851257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:46.851259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:46.851499 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:46.852437 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:48:46.852448 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:48:46.852775 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 15:48:46.853026 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 15:48:46.853579 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1823,"mutex_wait_us":119,"spinlock_wait_cycles":8704,"thread_start_us":1646,"threads_started":1}
I20260928 15:48:46.854131 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.830060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:34731 (local address 127.13.5.66:33071)
0928 15:48:46.832860 (+ 2800us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:46.832871 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:46.832902 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:48:46.834407 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:46.834416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:46.834820 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:46.835159 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:46.837180 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:46.838357 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:46.840574 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:46.841234 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:46.841457 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:46.845374 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:46.845382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:46.845398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:46.845437 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:46.849332 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:46.850825 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:46.850835 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:46.850844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:46.851104 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:46.851635 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:46.851640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:46.851645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:46.852279 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:46.852501 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:46.853286 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 15:48:46.853696 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2633,"thread_start_us":2533,"threads_started":1}
I20260928 15:48:46.854374 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.829928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:41981 (local address 127.13.5.67:34569)
0928 15:48:46.832562 (+ 2634us) server_negotiation.cc:206] Beginning negotiation
0928 15:48:46.832579 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:48:46.832632 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:48:46.834421 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:48:46.834431 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:48:46.834823 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:48:46.835185 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:48:46.837181 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:46.838473 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:46.840711 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:48:46.841435 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:48:46.841640 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:46.845735 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:48:46.845742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:48:46.845755 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:48:46.845798 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:48:46.849517 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:46.850825 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:46.850835 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:46.850844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:46.851083 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:48:46.851663 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:48:46.851670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:48:46.851674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:48:46.852384 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:48:46.852594 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 15:48:46.853522 (+ 928us) server_negotiation.cc:299] Negotiation successful
0928 15:48:46.853926 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2423,"thread_start_us":2281,"threads_started":1}
I20260928 15:48:46.855098 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.829691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:34569 (local address 127.13.5.65:41981)
0928 15:48:46.830126 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:46.830176 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:48:46.830909 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:48:46.835322 (+ 4413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:48:46.835334 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:48:46.836200 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:48:46.836955 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:46.836977 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:46.838736 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:46.838743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:46.840544 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:48:46.840559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:48:46.841681 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:48:46.841688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:48:46.841872 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:48:46.842619 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:48:46.842639 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:48:46.845365 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:48:46.849657 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:46.849661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:46.849677 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:46.850642 (+ 965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:46.851216 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:48:46.851221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:48:46.851224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:48:46.851506 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:48:46.852666 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:48:46.852673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:48:46.852841 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 15:48:46.853018 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 15:48:46.854762 (+ 1744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":1410,"spinlock_wait_cycles":137216}
W20260928 15:48:46.858296 30046 leader_election.cc:341] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569): Not found: Tablet not found: d86871864b1d4250a5545b8c29c85ff5
W20260928 15:48:46.858621 30048 leader_election.cc:341] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071): Not found: Tablet not found: d86871864b1d4250a5545b8c29c85ff5
I20260928 15:48:46.858795 30048 leader_election.cc:302] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb10e06365564c23bf1b422f34a2c03f; no voters: 9e2729c54e3b43ceaa01a9c6926d55b9, ad4bd7711d9a4b638edc27ccabd81875
I20260928 15:48:46.859267 30433 raft_consensus.cc:2726] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:48:46.863765 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:46.862884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:57579)
0928 15:48:46.863372 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:46.863521 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260928 15:48:47.348091 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:47.347102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:55537)
0928 15:48:47.347620 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:47.347765 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":233,"threads_started":1}
I20260928 15:48:47.398036 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:47.397315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:40859)
0928 15:48:47.397787 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:47.397871 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260928 15:48:47.865465 30440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:47.864782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:38099)
0928 15:48:47.865222 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:47.865308 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":212,"threads_started":1}
I20260928 15:48:48.349984 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:48.349140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:56773)
0928 15:48:48.349663 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:48.349779 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":285,"threads_started":1}
I20260928 15:48:48.400048 30442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:48.399076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:55769)
0928 15:48:48.399651 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:48.399744 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":276,"threads_started":1}
I20260928 15:48:48.618289 30443 raft_consensus.cc:482] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:48:48.618530 30443 raft_consensus.cc:504] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:48.619197 30443 leader_election.cc:288] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569), ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
W20260928 15:48:48.620338 30048 leader_election.cc:341] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071): Not found: Tablet not found: d86871864b1d4250a5545b8c29c85ff5
W20260928 15:48:48.620510 30046 leader_election.cc:341] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569): Not found: Tablet not found: d86871864b1d4250a5545b8c29c85ff5
I20260928 15:48:48.620631 30046 leader_election.cc:302] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb10e06365564c23bf1b422f34a2c03f; no voters: 9e2729c54e3b43ceaa01a9c6926d55b9, ad4bd7711d9a4b638edc27ccabd81875
I20260928 15:48:48.620869 30443 raft_consensus.cc:2726] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:48:48.867158 30444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:48.866383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:51157)
0928 15:48:48.866877 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:48.866987 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260928 15:48:49.351760 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:49.350936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:39887)
0928 15:48:49.351472 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:49.351559 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":291,"threads_started":1}
I20260928 15:48:49.401803 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:49.401035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:45767)
0928 15:48:49.401530 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:49.401629 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260928 15:48:49.868932 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:49.868146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:60037)
0928 15:48:49.868665 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:49.868747 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":244,"threads_started":1}
I20260928 15:48:50.353487 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:50.352712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:37809)
0928 15:48:50.353219 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:50.353301 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":259,"threads_started":1}
I20260928 15:48:50.404114 30449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:50.403222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:54207)
0928 15:48:50.403756 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:50.403853 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":308,"threads_started":1}
I20260928 15:48:50.735689 30450 raft_consensus.cc:482] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:48:50.736035 30450 raft_consensus.cc:504] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:50.736936 30450 leader_election.cc:288] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569), ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
W20260928 15:48:50.737884 30046 leader_election.cc:341] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569): Not found: Tablet not found: d86871864b1d4250a5545b8c29c85ff5
W20260928 15:48:50.738121 30048 leader_election.cc:341] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071): Not found: Tablet not found: d86871864b1d4250a5545b8c29c85ff5
I20260928 15:48:50.738247 30048 leader_election.cc:302] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb10e06365564c23bf1b422f34a2c03f; no voters: 9e2729c54e3b43ceaa01a9c6926d55b9, ad4bd7711d9a4b638edc27ccabd81875
I20260928 15:48:50.738485 30450 raft_consensus.cc:2726] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:48:50.870870 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:50.869984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:56699)
0928 15:48:50.870602 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:50.870705 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":284,"threads_started":1}
I20260928 15:48:51.355355 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:51.354542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:34813)
0928 15:48:51.355110 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:51.355191 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":272,"threads_started":1}
I20260928 15:48:51.405925 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:51.405142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:60149)
0928 15:48:51.405641 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:51.405747 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":251,"threads_started":1}
I20260928 15:48:51.872771 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:51.871989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:53119)
0928 15:48:51.872496 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:51.872587 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260928 15:48:52.357232 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:52.356370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:57635)
0928 15:48:52.356959 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:52.357047 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":349,"threads_started":1}
I20260928 15:48:52.407836 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:52.406950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:55955)
0928 15:48:52.407508 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:52.407617 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260928 15:48:52.874790 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:52.873915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:57915)
0928 15:48:52.874473 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:52.874596 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":230,"threads_started":1}
I20260928 15:48:53.359212 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:53.358279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:48653)
0928 15:48:53.358912 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:53.359003 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":324,"threads_started":1}
I20260928 15:48:53.409760 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:53.408984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:46661)
0928 15:48:53.409511 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:53.409599 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":322,"threads_started":1}
I20260928 15:48:53.876783 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:53.875816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:43339)
0928 15:48:53.876366 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:53.876524 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":323,"threads_started":1}
I20260928 15:48:54.361227 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:54.360431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:49977)
0928 15:48:54.360876 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:54.360992 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
I20260928 15:48:54.411764 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:54.410858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:45759)
0928 15:48:54.411469 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:54.411558 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":322,"threads_started":1}
I20260928 15:48:54.788255 30463 raft_consensus.cc:482] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:48:54.788511 30463 raft_consensus.cc:504] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2729c54e3b43ceaa01a9c6926d55b9" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 34569 } } peers { permanent_uuid: "eb10e06365564c23bf1b422f34a2c03f" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 41567 } } peers { permanent_uuid: "ad4bd7711d9a4b638edc27ccabd81875" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33071 } }
I20260928 15:48:54.789251 30463 leader_election.cc:288] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569), ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071)
W20260928 15:48:54.790390 30048 leader_election.cc:341] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad4bd7711d9a4b638edc27ccabd81875 (127.13.5.66:33071): Not found: Tablet not found: d86871864b1d4250a5545b8c29c85ff5
W20260928 15:48:54.790550 30046 leader_election.cc:341] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e2729c54e3b43ceaa01a9c6926d55b9 (127.13.5.67:34569): Not found: Tablet not found: d86871864b1d4250a5545b8c29c85ff5
I20260928 15:48:54.790691 30046 leader_election.cc:302] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb10e06365564c23bf1b422f34a2c03f; no voters: 9e2729c54e3b43ceaa01a9c6926d55b9, ad4bd7711d9a4b638edc27ccabd81875
I20260928 15:48:54.790961 30463 raft_consensus.cc:2726] T d86871864b1d4250a5545b8c29c85ff5 P eb10e06365564c23bf1b422f34a2c03f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 15:48:54.878603 30464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:54.877784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:47313)
0928 15:48:54.878276 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:54.878373 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260928 15:48:55.362922 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:55.362197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:58019)
0928 15:48:55.362669 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:55.362764 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":221,"threads_started":1}
I20260928 15:48:55.413568 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:55.412845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:42725)
0928 15:48:55.413314 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:55.413397 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260928 15:48:55.880512 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:55.879643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:45797)
0928 15:48:55.880210 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:55.880304 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":335,"threads_started":1}
I20260928 15:48:56.364905 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:56.363967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:39609)
0928 15:48:56.364633 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:56.364720 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":449,"threads_started":1}
I20260928 15:48:56.415411 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:56.414636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:58235)
0928 15:48:56.415138 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:56.415228 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":282,"threads_started":1}
I20260928 15:48:56.882342 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:56.881573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:52533)
0928 15:48:56.882088 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:56.882174 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":237,"threads_started":1}
I20260928 15:48:57.366611 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:57.365881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:56515)
0928 15:48:57.366329 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:57.366438 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
I20260928 15:48:57.417156 30472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:57.416388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:56515)
0928 15:48:57.416887 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:57.416979 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
I20260928 15:48:57.884186 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:57.883358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:53225)
0928 15:48:57.883799 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:57.883968 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
I20260928 15:48:58.368415 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:58.367639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:36399)
0928 15:48:58.368154 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:58.368241 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":286,"threads_started":1}
I20260928 15:48:58.419173 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:58.418294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:52651)
0928 15:48:58.418861 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:58.418969 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":342,"threads_started":1}
I20260928 15:48:58.886336 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:58.885450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:50145)
0928 15:48:58.886053 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:58.886165 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":269,"threads_started":1}
I20260928 15:48:59.370136 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:59.369389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:56091)
0928 15:48:59.369853 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:59.369947 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260928 15:48:59.420940 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:59.420160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:47947)
0928 15:48:59.420640 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:59.420737 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":237,"threads_started":1}
I20260928 15:48:59.888208 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:48:59.887303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:53691)
0928 15:48:59.887838 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 15:48:59.888012 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":243,"threads_started":1}
I20260928 15:49:00.372018 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:00.371070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:37151)
0928 15:49:00.371608 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:00.371703 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":296,"threads_started":1}
I20260928 15:49:00.423233 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:00.422315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:52709)
0928 15:49:00.422909 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:00.423040 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":339,"threads_started":1}
I20260928 15:49:00.890380 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:00.889576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.66:43009)
0928 15:49:00.890045 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:00.890169 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":220,"threads_started":1}
I20260928 15:49:01.347648 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 29990
I20260928 15:49:01.367805 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:01.349029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.13.5.65:41567)
0928 15:49:01.349631 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:01.349644 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:01.349679 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:49:01.349771 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:01.349783 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:01.350109 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:01.350288 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:01.351782 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:01.352821 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:01.354046 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:01.354599 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:01.354700 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:01.361975 (+ 7275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:01.361989 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:01.362041 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:01.362108 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:01.365219 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:01.365951 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:01.365958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:01.365961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:01.366088 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:01.366408 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:01.366411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:01.366413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:01.366767 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:01.367024 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:01.367261 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 15:49:01.367545 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":1056,"threads_started":1}
I20260928 15:49:01.369226 30157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:01.388546 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:01.373226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.67:48205)
0928 15:49:01.388230 (+ 15004us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:01.388358 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14866,"threads_started":1}
I20260928 15:49:01.468881 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 29990
I20260928 15:49:01.468971 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:01.461028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:45959 (local address 127.13.5.65:48485)
0928 15:49:01.468432 (+ 7404us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:01.468648 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.126:45959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7268,"thread_start_us":7090,"threads_started":1}
I20260928 15:49:01.489149 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 30166
I20260928 15:49:01.503312 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:01.490086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.13.5.66:33071)
0928 15:49:01.490778 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:01.490782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:01.490799 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:49:01.490856 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:01.490859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:01.490959 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:01.491106 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:01.492587 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:01.493390 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:01.494760 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:01.495205 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:01.495308 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:01.498062 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:01.498067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:01.498070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:01.498103 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:01.500785 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:01.501470 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:01.501474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:01.501476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:01.501628 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:01.501954 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:01.501957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:01.501959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:01.502226 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:01.502413 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:01.502678 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 15:49:01.502845 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":379,"threads_started":1}
I20260928 15:49:01.504076 30290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:01.590384 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 30166
I20260928 15:49:01.609591 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 30298
I20260928 15:49:01.624641 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:01.610442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.13.5.67:34569)
0928 15:49:01.611062 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:01.611067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:01.611090 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:49:01.611148 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:01.611152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:01.611214 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:01.611304 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:01.612911 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:01.613770 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:01.615087 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:01.615576 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:01.615682 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:01.618766 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:01.618771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:01.618774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:01.618812 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:01.621787 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:01.622540 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:01.622546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:01.622550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:01.622738 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:01.623102 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:01.623105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:01.623107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:01.623383 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:01.623597 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:01.624119 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 15:49:01.624311 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":306,"threads_started":1}
I20260928 15:49:01.625403 30423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:01.712641 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 30298
2026-09-28T15:49:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64924 ms)
[----------] 30 tests from SecurityITest (302999 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:01 dist-test-slave-j968 krb5kdc[30493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:01 dist-test-slave-j968 krb5kdc[30493](info): set up 2 sockets
Sep 28 15:49:01 dist-test-slave-j968 krb5kdc[30493](info): commencing operation
krb5kdc: starting...
W20260928 15:49:02.835247 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:03 dist-test-slave-j968 krb5kdc[30493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:03.096828 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:40477
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:40477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:03.457450 30506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:03.457748 30506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:03.457912 30506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:03.465837 30506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:03.465960 30506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:03.466064 30506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:03.466199 30506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:03.466249 30506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:03.476490 30506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:40477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:40477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:03.477888 30506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:03.479264 30506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:03.491770 30512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:03.491734 30514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:03.491786 30511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:03.492127 30506 server_base.cc:1034] running on GCE node
I20260928 15:49:03.493228 30506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:03.494671 30506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:03.495966 30506 hybrid_clock.cc:648] HybridClock initialized: now 1790610543495828 us; error 55 us; skew 500 ppm
I20260928 15:49:03.496338 30506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:03 dist-test-slave-j968 krb5kdc[30493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610543, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:03.499432 30506 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:49:03.501106 30506 webserver.cc:460] Webserver started at http://127.13.5.126:37527/ using document root <none> and password file <none>
I20260928 15:49:03.501945 30506 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:03.502079 30506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:03.502442 30506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:03.505678 30506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "c71a74dc8b0240a29e834b50b26c6f24"
format_stamp: "Formatted at 2026-09-28 15:49:03 on dist-test-slave-j968"
server_key: "667a0429b66142ea5f4df0ce207b68fd"
server_key_iv: "b465fc37238fcf601f4bc350c367ba66"
server_key_version: "encryptionkey@0"
I20260928 15:49:03.506520 30506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "c71a74dc8b0240a29e834b50b26c6f24"
format_stamp: "Formatted at 2026-09-28 15:49:03 on dist-test-slave-j968"
server_key: "667a0429b66142ea5f4df0ce207b68fd"
server_key_iv: "b465fc37238fcf601f4bc350c367ba66"
server_key_version: "encryptionkey@0"
I20260928 15:49:03.512087 30506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:49:03.515566 30521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:03.517047 30506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:03.517246 30506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "c71a74dc8b0240a29e834b50b26c6f24"
format_stamp: "Formatted at 2026-09-28 15:49:03 on dist-test-slave-j968"
server_key: "667a0429b66142ea5f4df0ce207b68fd"
server_key_iv: "b465fc37238fcf601f4bc350c367ba66"
server_key_version: "encryptionkey@0"
I20260928 15:49:03.517454 30506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:03.531143 30506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:03.536058 30506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:03.536352 30506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:03.556154 30506 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:40477
I20260928 15:49:03.556196 30572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:40477 every 8 connection(s)
I20260928 15:49:03.557649 30506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:03.561821 30573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:03.563369 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 30506
I20260928 15:49:03.563618 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:03.564585 13333 external_mini_cluster.cc:1428] Setting key 4c502e039c4b68c07567dae40a5142d7
I20260928 15:49:03.578604 30573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24: Bootstrap starting.
Sep 28 15:49:03 dist-test-slave-j968 krb5kdc[30493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:03.582753 30573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:03.584035 30573 log.cc:824] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:03.588063 30573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24: No bootstrap required, opened a new log
I20260928 15:49:03.591324 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:03.566868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.13.5.126:40477)
0928 15:49:03.567756 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:03.567772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:03.567826 (+ 54us) server_negotiation.cc:407] Connection header received
0928 15:49:03.569543 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:03.569572 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:03.569987 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:03.570333 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:03.572455 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:03.573828 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:03.576034 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:03.576833 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:03.576988 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:03.582237 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:03.582245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:03.582261 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:03.582307 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:03.586174 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:03.586964 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:03.586977 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:03.586989 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:03.587222 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:03.587663 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:03.587668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:03.587672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:03.588487 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:03.588842 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:03.589259 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 15:49:03.589763 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":412,"threads_started":1}
I20260928 15:49:03.598548 30573 raft_consensus.cc:348] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a74dc8b0240a29e834b50b26c6f24" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40477 } }
I20260928 15:49:03.598984 30573 raft_consensus.cc:374] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:03.599066 30573 raft_consensus.cc:729] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71a74dc8b0240a29e834b50b26c6f24, State: Initialized, Role: FOLLOWER
I20260928 15:49:03.599726 30573 consensus_queue.cc:260] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a74dc8b0240a29e834b50b26c6f24" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40477 } }
I20260928 15:49:03.600049 30573 raft_consensus.cc:388] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:03.600193 30573 raft_consensus.cc:482] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:03.600376 30573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:03.602916 30573 raft_consensus.cc:504] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a74dc8b0240a29e834b50b26c6f24" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40477 } }
I20260928 15:49:03.603505 30573 leader_election.cc:302] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c71a74dc8b0240a29e834b50b26c6f24; no voters:
I20260928 15:49:03.604302 30573 leader_election.cc:288] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:03.604435 30578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:03.605258 30578 raft_consensus.cc:686] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [term 1 LEADER]: Becoming Leader. State: Replica: c71a74dc8b0240a29e834b50b26c6f24, State: Running, Role: LEADER
I20260928 15:49:03.605865 30578 consensus_queue.cc:237] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a74dc8b0240a29e834b50b26c6f24" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40477 } }
I20260928 15:49:03.606779 30573 sys_catalog.cc:564] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:03.608768 30580 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c71a74dc8b0240a29e834b50b26c6f24. Latest consensus state: current_term: 1 leader_uuid: "c71a74dc8b0240a29e834b50b26c6f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a74dc8b0240a29e834b50b26c6f24" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40477 } } }
I20260928 15:49:03.609262 30580 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:03.610263 30579 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c71a74dc8b0240a29e834b50b26c6f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71a74dc8b0240a29e834b50b26c6f24" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 40477 } } }
I20260928 15:49:03.610623 30579 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:03.612738 30585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:03.617923 30585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:03.629228 30585 catalog_manager.cc:1347] Generated new cluster ID: b9efceaaf26b489098ff5035c7d4764d
I20260928 15:49:03.629366 30585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:03.650861 30585 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:03.652865 30585 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:03.666530 30585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24: Generated new TSK 0
I20260928 15:49:03.667630 30585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:49:03.732177 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40477
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:04.084595 30599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:04.084899 30599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:04.084995 30599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:04.092515 30599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:04.092621 30599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:04.092733 30599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:49:04.102790 30599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:04.104122 30599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:04.105414 30599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:04.118992 30606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:04.119014 30608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:04.119254 30605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:04.119858 30599 server_base.cc:1034] running on GCE node
I20260928 15:49:04.120533 30599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:04.121680 30599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:04.122951 30599 hybrid_clock.cc:648] HybridClock initialized: now 1790610544122894 us; error 55 us; skew 500 ppm
I20260928 15:49:04.123304 30599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:04 dist-test-slave-j968 krb5kdc[30493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610544, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:04.126547 30599 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:49:04.128153 30599 webserver.cc:460] Webserver started at http://127.13.5.65:36979/ using document root <none> and password file <none>
I20260928 15:49:04.128818 30599 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:04.128932 30599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:04.129201 30599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:04.132496 30599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "03a59f5dd14a4a07ae93f263e4704a00"
format_stamp: "Formatted at 2026-09-28 15:49:04 on dist-test-slave-j968"
server_key: "cf6eefbcf642ca0358c6edd7849564fa"
server_key_iv: "51b70a0f1a943f3074d9f10f236307e1"
server_key_version: "encryptionkey@0"
I20260928 15:49:04.133625 30599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "03a59f5dd14a4a07ae93f263e4704a00"
format_stamp: "Formatted at 2026-09-28 15:49:04 on dist-test-slave-j968"
server_key: "cf6eefbcf642ca0358c6edd7849564fa"
server_key_iv: "51b70a0f1a943f3074d9f10f236307e1"
server_key_version: "encryptionkey@0"
I20260928 15:49:04.140643 30599 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:49:04.144464 30615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:04.145926 30599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:49:04.146090 30599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "03a59f5dd14a4a07ae93f263e4704a00"
format_stamp: "Formatted at 2026-09-28 15:49:04 on dist-test-slave-j968"
server_key: "cf6eefbcf642ca0358c6edd7849564fa"
server_key_iv: "51b70a0f1a943f3074d9f10f236307e1"
server_key_version: "encryptionkey@0"
I20260928 15:49:04.146271 30599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:04.165463 30599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:04.170130 30599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:04.170430 30599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:04.171460 30599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:04.172884 30599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:04.172958 30599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:04.173135 30599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:04.173194 30599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:04.207224 30599 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:39455
I20260928 15:49:04.207329 30730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:39455 every 8 connection(s)
I20260928 15:49:04.208642 30599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:49:04.209087 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 30599
I20260928 15:49:04.209321 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:49:04.210220 13333 external_mini_cluster.cc:1428] Setting key e544c596dc68e02972ecc7fdaebf4ed0
Sep 28 15:49:04 dist-test-slave-j968 krb5kdc[30493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610544, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:04.230255 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:04.211409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:50723 (local address 127.13.5.126:40477)
0928 15:49:04.211977 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:04.211983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:04.213138 (+ 1155us) server_negotiation.cc:407] Connection header received
0928 15:49:04.215017 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:04.215026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:04.215118 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:04.215237 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:04.217057 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:04.218262 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:04.219832 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:04.220427 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:04.220589 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:04.224382 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:04.224395 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:04.224398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:04.224432 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:04.227142 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:04.227899 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:04.227905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:04.227909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:04.228035 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:04.228475 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:04.228480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:04.228483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:04.228764 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:04.228922 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:04.229659 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 15:49:04.229894 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":291,"threads_started":1}
I20260928 15:49:04.231244 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:04.212117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40477 (local address 127.13.5.65:50723)
0928 15:49:04.212958 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:04.213006 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:04.214743 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:04.215582 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:04.215591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:04.216243 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:04.216888 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:04.216906 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:04.218415 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:04.218418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:04.219681 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:04.219693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:04.220682 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:04.220692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:04.220944 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:04.221682 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:04.221710 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:04.224154 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:04.227303 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:04.227312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:04.227330 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:04.227747 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:04.228147 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:04.228153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:04.228156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:04.228342 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:04.228895 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:04.228902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:04.229251 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 15:49:04.229549 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 15:49:04.230098 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":362,"threads_started":1}
I20260928 15:49:04.232945 30731 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40477
I20260928 15:49:04.233322 30731 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:04.234305 30731 heartbeater.cc:511] Master 127.13.5.126:40477 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
I20260928 15:49:04.236858 30538 ts_manager.cc:194] Registered new tserver with Master: 03a59f5dd14a4a07ae93f263e4704a00 (127.13.5.65:39455)
I20260928 15:49:04.238287 30538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:50723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:49:04.270828 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40477
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:04.602934 30737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:04.603300 30737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:04.603471 30737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:04.610829 30737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:04.610980 30737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:04.611173 30737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:49:04.621279 30737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:04.622705 30737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:04.624300 30737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:04.638092 30743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:04.638178 30742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:04.638159 30745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:04.639030 30737 server_base.cc:1034] running on GCE node
I20260928 15:49:04.639672 30737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:04.640684 30737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:04.641893 30737 hybrid_clock.cc:648] HybridClock initialized: now 1790610544641852 us; error 33 us; skew 500 ppm
I20260928 15:49:04.642196 30737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:04 dist-test-slave-j968 krb5kdc[30493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610544, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:04.645604 30737 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:49:04.647190 30737 webserver.cc:460] Webserver started at http://127.13.5.66:40773/ using document root <none> and password file <none>
I20260928 15:49:04.647953 30737 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:04.648077 30737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:04.648352 30737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:04.651244 30737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "04fad925b878425c9f84fe809323e8fe"
format_stamp: "Formatted at 2026-09-28 15:49:04 on dist-test-slave-j968"
server_key: "a2718d84311f5339d069e28a50b62857"
server_key_iv: "097d75ef3e655e987221d6039f8dfee8"
server_key_version: "encryptionkey@0"
I20260928 15:49:04.652036 30737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "04fad925b878425c9f84fe809323e8fe"
format_stamp: "Formatted at 2026-09-28 15:49:04 on dist-test-slave-j968"
server_key: "a2718d84311f5339d069e28a50b62857"
server_key_iv: "097d75ef3e655e987221d6039f8dfee8"
server_key_version: "encryptionkey@0"
I20260928 15:49:04.657258 30737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:49:04.660918 30752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:04.662325 30737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:04.662462 30737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "04fad925b878425c9f84fe809323e8fe"
format_stamp: "Formatted at 2026-09-28 15:49:04 on dist-test-slave-j968"
server_key: "a2718d84311f5339d069e28a50b62857"
server_key_iv: "097d75ef3e655e987221d6039f8dfee8"
server_key_version: "encryptionkey@0"
I20260928 15:49:04.662631 30737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:04.683987 30737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:04.688519 30737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:04.688759 30737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:04.689723 30737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:04.691099 30737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:04.691174 30737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:04.691262 30737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:04.691305 30737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:04.726293 30737 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:33725
I20260928 15:49:04.726445 30866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:33725 every 8 connection(s)
I20260928 15:49:04.727679 30737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:49:04.735702 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 30737
I20260928 15:49:04.735926 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:49:04.736418 13333 external_mini_cluster.cc:1428] Setting key 885ba7ae1b357913fa43c8a07a9c027d
Sep 28 15:49:04 dist-test-slave-j968 krb5kdc[30493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610544, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:04.749269 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:04.730538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51483 (local address 127.13.5.126:40477)
0928 15:49:04.730771 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:04.730776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:04.732436 (+ 1660us) server_negotiation.cc:407] Connection header received
0928 15:49:04.734481 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:04.734486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:04.734566 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:04.734662 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:04.736567 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:04.737430 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:04.738999 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:04.739461 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:04.739598 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:04.743403 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:04.743408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:04.743413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:04.743450 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:04.746133 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:04.746873 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:04.746883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:04.746886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:04.747016 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:04.747483 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:04.747487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:04.747489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:04.747765 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:04.747952 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:04.748652 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 15:49:04.748899 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 15:49:04.750383 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:04.731134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40477 (local address 127.13.5.66:51483)
0928 15:49:04.732212 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:04.732279 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:04.734186 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:04.734897 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:04.734906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:04.735590 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:04.736365 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:04.736384 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:04.737578 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:04.737583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:04.738835 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:04.738846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:04.739696 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:04.739703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:04.740000 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:04.740687 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:04.740715 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:04.743166 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:04.746277 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:04.746287 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:04.746304 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:04.746745 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:04.747125 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:04.747130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:04.747134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:04.747334 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:04.747910 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:04.747919 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:04.748270 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 15:49:04.748546 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 15:49:04.749096 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":487,"threads_started":1}
I20260928 15:49:04.751978 30867 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40477
I20260928 15:49:04.752288 30867 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:04.753019 30867 heartbeater.cc:511] Master 127.13.5.126:40477 requested a full tablet report, sending...
I20260928 15:49:04.754545 30538 ts_manager.cc:194] Registered new tserver with Master: 04fad925b878425c9f84fe809323e8fe (127.13.5.66:33725)
I20260928 15:49:04.755846 30538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:51483
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:49:04.793146 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40477
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:39727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:05.133219 30872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:05.133517 30872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:05.133616 30872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:05.141062 30872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:05.141211 30872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:05.141325 30872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:49:05.151469 30872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:39727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:40477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:05.152774 30872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:05.154070 30872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:05.168120 30877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:05.168053 30880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:05.168089 30878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:05.169093 30872 server_base.cc:1034] running on GCE node
I20260928 15:49:05.169740 30872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:05.170718 30872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:05.172000 30872 hybrid_clock.cc:648] HybridClock initialized: now 1790610545171891 us; error 77 us; skew 500 ppm
I20260928 15:49:05.172384 30872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:05 dist-test-slave-j968 krb5kdc[30493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610545, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:05.175525 30872 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:49:05.177284 30872 webserver.cc:460] Webserver started at http://127.13.5.67:42413/ using document root <none> and password file <none>
I20260928 15:49:05.177986 30872 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:05.178144 30872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:05.178457 30872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:05.181481 30872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "625ead7f391a474ca812397093a3d65f"
format_stamp: "Formatted at 2026-09-28 15:49:05 on dist-test-slave-j968"
server_key: "1442d622a808c353692023ea898118e6"
server_key_iv: "cd5622de0dfe239ffa4e922f44452fd1"
server_key_version: "encryptionkey@0"
I20260928 15:49:05.182287 30872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "625ead7f391a474ca812397093a3d65f"
format_stamp: "Formatted at 2026-09-28 15:49:05 on dist-test-slave-j968"
server_key: "1442d622a808c353692023ea898118e6"
server_key_iv: "cd5622de0dfe239ffa4e922f44452fd1"
server_key_version: "encryptionkey@0"
I20260928 15:49:05.187757 30872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 15:49:05.191356 30887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:05.192909 30872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:05.193104 30872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "625ead7f391a474ca812397093a3d65f"
format_stamp: "Formatted at 2026-09-28 15:49:05 on dist-test-slave-j968"
server_key: "1442d622a808c353692023ea898118e6"
server_key_iv: "cd5622de0dfe239ffa4e922f44452fd1"
server_key_version: "encryptionkey@0"
I20260928 15:49:05.193326 30872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:05.211863 30872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:05.216473 30872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:05.216773 30872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:05.217851 30872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:05.219240 30872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:05.219344 30872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:05.219503 30872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:05.219583 30872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:05.241808 30731 heartbeater.cc:503] Master 127.13.5.126:40477 was elected leader, sending a full tablet report...
I20260928 15:49:05.254644 30872 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:42551
I20260928 15:49:05.254691 31001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:42551 every 8 connection(s)
I20260928 15:49:05.256021 30872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:49:05.258802 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 30872
I20260928 15:49:05.258985 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:49:05.259562 13333 external_mini_cluster.cc:1428] Setting key 3e68fc088222e979430a09c0a3ab32cc
Sep 28 15:49:05 dist-test-slave-j968 krb5kdc[30493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610545, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:05.277838 31004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.258548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:48637 (local address 127.13.5.126:40477)
0928 15:49:05.259088 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.259096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.260354 (+ 1258us) server_negotiation.cc:407] Connection header received
0928 15:49:05.262477 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.262483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.262570 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.262687 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:05.264341 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.265180 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.267387 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.267825 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.268014 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.271538 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:05.271543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:05.271546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:05.271579 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:05.274645 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:05.275657 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:05.275661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:05.275663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:05.275784 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:05.276319 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:05.276322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:05.276324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:05.276597 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:05.276707 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.277402 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.277576 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
I20260928 15:49:05.278855 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.259269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:40477 (local address 127.13.5.67:48637)
0928 15:49:05.260132 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:05.260208 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:05.262217 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:05.262971 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:05.262988 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:05.263512 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:05.264176 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.264195 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.265603 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.265611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.267213 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.267227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.268107 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.268113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.268332 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.269014 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:05.269070 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:05.271341 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:05.274800 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:05.274809 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:05.274833 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:05.275547 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:05.276001 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:05.276005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:05.276008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:05.276198 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:05.276723 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:05.276729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:05.277063 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 15:49:05.277297 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 15:49:05.277764 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":378,"threads_started":1}
I20260928 15:49:05.280273 31002 heartbeater.cc:347] Connected to a master server at 127.13.5.126:40477
I20260928 15:49:05.280599 31002 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:05.281361 31002 heartbeater.cc:511] Master 127.13.5.126:40477 requested a full tablet report, sending...
I20260928 15:49:05.282861 30538 ts_manager.cc:194] Registered new tserver with Master: 625ead7f391a474ca812397093a3d65f (127.13.5.67:42551)
I20260928 15:49:05.283720 30538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:48637
I20260928 15:49:05.287683 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:49:05.305454 31004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.292268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.13.5.126:40477)
0928 15:49:05.292427 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.292431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.292966 (+ 535us) server_negotiation.cc:407] Connection header received
0928 15:49:05.293092 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.293096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.293160 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.293256 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:05.294761 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.295583 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.297004 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.297527 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.297654 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.299974 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:05.299980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:05.299983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:05.300016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:05.302495 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:05.303158 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:05.303161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:05.303164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:05.303313 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:05.303791 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:05.303799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:05.303803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:05.304210 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:05.304579 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.304854 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.305097 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:49:05.309893 30538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:49:05.317116 30538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:49:05.346777 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.330892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.13.5.65:39455)
0928 15:49:05.332516 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.332526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.332557 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:49:05.332672 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.332684 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.333055 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.333345 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:05.334699 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.337170 (+ 2471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.341626 (+ 4456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.344222 (+ 2596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.344394 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.344742 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.346185 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.346394 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1444,"thread_start_us":1281,"threads_started":1}
I20260928 15:49:05.347151 31015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.330686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39455 (local address 127.0.0.1:49384)
0928 15:49:05.331371 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:05.331410 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:05.331559 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:05.333776 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:05.333789 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:05.333835 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:05.334351 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.334376 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.337410 (+ 3034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.337421 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.341287 (+ 3866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.341313 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.344907 (+ 3594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.344914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.345173 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.345247 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 15:49:05.345405 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 15:49:05.345649 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":346,"threads_started":1}
I20260928 15:49:05.351910 30664 tablet_service.cc:1467] Processing CreateTablet for tablet 97b5359f8dc64ac5a8fffa0022c5a96c (DEFAULT_TABLE table=test-table [id=b49d0604cbcd47119d857836e157a5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:05.353650 30664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97b5359f8dc64ac5a8fffa0022c5a96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:05.354206 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.332551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:42551 (local address 127.0.0.1:36468)
0928 15:49:05.344762 (+ 12211us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:05.344787 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:05.344933 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:05.346538 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:05.346547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:05.346587 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:05.347286 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.347305 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.349836 (+ 2531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.349842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.351958 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.351972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.353369 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.353375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.353543 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.353578 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:49:05.353681 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:49:05.353816 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12037,"mutex_wait_us":38,"thread_start_us":12478,"threads_started":1}
I20260928 15:49:05.354950 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.332378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.13.5.67:42551)
0928 15:49:05.345780 (+ 13402us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.345790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.345830 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:49:05.345957 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.345966 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.346282 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.346522 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:05.347891 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.349654 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.352132 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.353166 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.353323 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.353577 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.354443 (+ 866us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.354612 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13198,"thread_start_us":13386,"threads_started":1}
I20260928 15:49:05.355981 31015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.332633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33725 (local address 127.0.0.1:38284)
0928 15:49:05.347475 (+ 14842us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:05.347494 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:05.347636 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:05.348418 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:05.348422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:05.348446 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:05.348652 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.348665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.351100 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.351105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.353034 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.353050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.355000 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.355006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.355307 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.355333 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:49:05.355453 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 15:49:05.355596 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14708,"mutex_wait_us":26,"thread_start_us":15007,"threads_started":1}
I20260928 15:49:05.356108 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.333224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38284 (local address 127.13.5.66:33725)
0928 15:49:05.342654 (+ 9430us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.342667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.347594 (+ 4927us) server_negotiation.cc:407] Connection header received
0928 15:49:05.348002 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.348012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.348228 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.348437 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:05.348838 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.350923 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.353235 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.354833 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.354988 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.355195 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.355501 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.355689 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9204,"thread_start_us":9058,"threads_started":1}
I20260928 15:49:05.358991 30937 tablet_service.cc:1467] Processing CreateTablet for tablet 97b5359f8dc64ac5a8fffa0022c5a96c (DEFAULT_TABLE table=test-table [id=b49d0604cbcd47119d857836e157a5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:05.359959 30802 tablet_service.cc:1467] Processing CreateTablet for tablet 97b5359f8dc64ac5a8fffa0022c5a96c (DEFAULT_TABLE table=test-table [id=b49d0604cbcd47119d857836e157a5be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:05.360603 30937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97b5359f8dc64ac5a8fffa0022c5a96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:05.361327 30802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97b5359f8dc64ac5a8fffa0022c5a96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:05.368579 31021 tablet_bootstrap.cc:492] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Bootstrap starting.
I20260928 15:49:05.372745 31021 tablet_bootstrap.cc:654] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:05.374639 31022 tablet_bootstrap.cc:492] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Bootstrap starting.
I20260928 15:49:05.374714 31021 log.cc:824] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:05.377205 31024 tablet_bootstrap.cc:492] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Bootstrap starting.
I20260928 15:49:05.378571 31022 tablet_bootstrap.cc:654] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:05.378753 31021 tablet_bootstrap.cc:492] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: No bootstrap required, opened a new log
I20260928 15:49:05.379073 31021 ts_tablet_manager.cc:1397] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 15:49:05.379791 31022 log.cc:824] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:05.380870 31024 tablet_bootstrap.cc:654] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:05.382155 31024 log.cc:824] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:05.382979 31022 tablet_bootstrap.cc:492] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: No bootstrap required, opened a new log
I20260928 15:49:05.383350 31022 ts_tablet_manager.cc:1397] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260928 15:49:05.385617 31024 tablet_bootstrap.cc:492] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: No bootstrap required, opened a new log
I20260928 15:49:05.385913 31024 ts_tablet_manager.cc:1397] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:49:05.388447 31021 raft_consensus.cc:348] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.388828 31021 raft_consensus.cc:374] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:05.388986 31021 raft_consensus.cc:729] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a59f5dd14a4a07ae93f263e4704a00, State: Initialized, Role: FOLLOWER
I20260928 15:49:05.389654 31021 consensus_queue.cc:260] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.391578 31021 ts_tablet_manager.cc:1428] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 15:49:05.395519 31022 raft_consensus.cc:348] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.396080 31022 raft_consensus.cc:374] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:05.396198 31022 raft_consensus.cc:729] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625ead7f391a474ca812397093a3d65f, State: Initialized, Role: FOLLOWER
I20260928 15:49:05.396968 31022 consensus_queue.cc:260] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.398525 31002 heartbeater.cc:503] Master 127.13.5.126:40477 was elected leader, sending a full tablet report...
I20260928 15:49:05.398269 31024 raft_consensus.cc:348] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.398778 31024 raft_consensus.cc:374] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:05.398895 31024 raft_consensus.cc:729] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fad925b878425c9f84fe809323e8fe, State: Initialized, Role: FOLLOWER
I20260928 15:49:05.399719 31024 consensus_queue.cc:260] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.400981 30867 heartbeater.cc:503] Master 127.13.5.126:40477 was elected leader, sending a full tablet report...
I20260928 15:49:05.401081 31022 ts_tablet_manager.cc:1428] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260928 15:49:05.401798 31024 ts_tablet_manager.cc:1428] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260928 15:49:05.461257 30732 tablet.cc:2369] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:49:05.479912 30868 tablet.cc:2369] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:49:05.507750 31003 tablet.cc:2369] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:49:05.721673 31029 raft_consensus.cc:482] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:49:05.722025 31029 raft_consensus.cc:504] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.723830 31029 leader_election.cc:288] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03a59f5dd14a4a07ae93f263e4704a00 (127.13.5.65:39455), 625ead7f391a474ca812397093a3d65f (127.13.5.67:42551)
I20260928 15:49:05.732136 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.724393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:45763 (local address 127.13.5.67:42551)
0928 15:49:05.724622 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.724631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.725333 (+ 702us) server_negotiation.cc:407] Connection header received
0928 15:49:05.725551 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.725556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.725632 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.725719 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:05.726530 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.727622 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.730151 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.731107 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.731266 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.731382 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.731716 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.731858 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:49:05.732296 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.724254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:42551 (local address 127.13.5.66:45763)
0928 15:49:05.725215 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:05.725229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:05.725378 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:05.725758 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:05.725762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:05.725794 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:05.726324 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.726356 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.727780 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.727786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.729982 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.730023 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.731336 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.731342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.731526 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.731553 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 15:49:05.731685 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 15:49:05.731826 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":23,"thread_start_us":607,"threads_started":1}
I20260928 15:49:05.732427 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.724193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:52675 (local address 127.13.5.65:39455)
0928 15:49:05.724417 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.724425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.725107 (+ 682us) server_negotiation.cc:407] Connection header received
0928 15:49:05.725344 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.725349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.725411 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.725563 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:05.726434 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.727197 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.729992 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.731275 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.731403 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.731505 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.732051 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.732186 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:49:05.733204 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.724184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:39455 (local address 127.13.5.66:52675)
0928 15:49:05.724979 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:05.725002 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:05.725186 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:05.725608 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:05.725613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:05.725645 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:05.726259 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.726271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.727329 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.727337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.729795 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:05.729811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.731660 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.731666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:05.731835 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.731892 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:49:05.732669 (+ 777us) client_negotiation.cc:240] Negotiation successful
0928 15:49:05.732813 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":1,"thread_start_us":446,"threads_started":1}
I20260928 15:49:05.733459 30957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b5359f8dc64ac5a8fffa0022c5a96c" candidate_uuid: "04fad925b878425c9f84fe809323e8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625ead7f391a474ca812397093a3d65f" is_pre_election: true
I20260928 15:49:05.734083 30957 raft_consensus.cc:2454] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04fad925b878425c9f84fe809323e8fe in term 0.
I20260928 15:49:05.734222 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b5359f8dc64ac5a8fffa0022c5a96c" candidate_uuid: "04fad925b878425c9f84fe809323e8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a59f5dd14a4a07ae93f263e4704a00" is_pre_election: true
I20260928 15:49:05.734763 30684 raft_consensus.cc:2454] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04fad925b878425c9f84fe809323e8fe in term 0.
I20260928 15:49:05.734853 30753 leader_election.cc:302] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fad925b878425c9f84fe809323e8fe, 625ead7f391a474ca812397093a3d65f; no voters:
I20260928 15:49:05.735319 31029 raft_consensus.cc:2781] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:49:05.735497 31029 raft_consensus.cc:482] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:49:05.735637 31029 raft_consensus.cc:3037] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:05.738015 31029 raft_consensus.cc:504] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.738679 31029 leader_election.cc:288] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [CANDIDATE]: Term 1 election: Requested vote from peers 03a59f5dd14a4a07ae93f263e4704a00 (127.13.5.65:39455), 625ead7f391a474ca812397093a3d65f (127.13.5.67:42551)
I20260928 15:49:05.739130 30684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b5359f8dc64ac5a8fffa0022c5a96c" candidate_uuid: "04fad925b878425c9f84fe809323e8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a59f5dd14a4a07ae93f263e4704a00"
I20260928 15:49:05.739233 30957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b5359f8dc64ac5a8fffa0022c5a96c" candidate_uuid: "04fad925b878425c9f84fe809323e8fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625ead7f391a474ca812397093a3d65f"
I20260928 15:49:05.739334 30684 raft_consensus.cc:3037] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:05.739442 30957 raft_consensus.cc:3037] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:05.741717 30684 raft_consensus.cc:2454] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04fad925b878425c9f84fe809323e8fe in term 1.
I20260928 15:49:05.742002 30957 raft_consensus.cc:2454] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04fad925b878425c9f84fe809323e8fe in term 1.
I20260928 15:49:05.742229 30756 leader_election.cc:302] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a59f5dd14a4a07ae93f263e4704a00, 04fad925b878425c9f84fe809323e8fe; no voters:
I20260928 15:49:05.742635 31029 raft_consensus.cc:2781] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:05.743142 31029 raft_consensus.cc:686] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 1 LEADER]: Becoming Leader. State: Replica: 04fad925b878425c9f84fe809323e8fe, State: Running, Role: LEADER
I20260928 15:49:05.743634 31029 consensus_queue.cc:237] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } }
I20260928 15:49:05.748682 30537 catalog_manager.cc:5495] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 04fad925b878425c9f84fe809323e8fe (127.13.5.66). New cstate: current_term: 1 leader_uuid: "04fad925b878425c9f84fe809323e8fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04fad925b878425c9f84fe809323e8fe" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33725 } health_report { overall_health: HEALTHY } } }
I20260928 15:49:05.802071 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.796504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.13.5.66:33725)
0928 15:49:05.796727 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.796733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.796752 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:49:05.797023 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.797028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.797109 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.797250 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:49:05.797690 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.798456 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.799842 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.800302 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.800408 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.800873 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:49:05.801458 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:49:05.801584 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.801684 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.801752 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 15:49:05.810982 30957 raft_consensus.cc:1261] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 1 FOLLOWER]: Refusing update from remote peer 04fad925b878425c9f84fe809323e8fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:05.810982 30684 raft_consensus.cc:1261] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 1 FOLLOWER]: Refusing update from remote peer 04fad925b878425c9f84fe809323e8fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:05.811973 31029 consensus_queue.cc:1035] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a59f5dd14a4a07ae93f263e4704a00" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:05.812289 31032 consensus_queue.cc:1035] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "625ead7f391a474ca812397093a3d65f" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:05.848358 30537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:05.848644 30537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:05.852401 30537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c71a74dc8b0240a29e834b50b26c6f24: Sending DeleteTablet for 3 replicas of tablet 97b5359f8dc64ac5a8fffa0022c5a96c
I20260928 15:49:05.853650 30937 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b5359f8dc64ac5a8fffa0022c5a96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:05 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:36468
I20260928 15:49:05.853776 30802 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b5359f8dc64ac5a8fffa0022c5a96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:05 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:38284
I20260928 15:49:05.853924 30664 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b5359f8dc64ac5a8fffa0022c5a96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:05 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:49384
I20260928 15:49:05.856994 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 30599
I20260928 15:49:05.866259 31046 tablet_replica.cc:330] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: stopping tablet replica
I20260928 15:49:05.866959 31046 raft_consensus.cc:2229] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:49:05.867828 31046 raft_consensus.cc:2258] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:05.870328 31046 ts_tablet_manager.cc:1905] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:05.878291 31046 ts_tablet_manager.cc:1918] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:05.878541 31046 log.cc:1196] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/97b5359f8dc64ac5a8fffa0022c5a96c
I20260928 15:49:05.879138 31046 ts_tablet_manager.cc:1939] T 97b5359f8dc64ac5a8fffa0022c5a96c P 04fad925b878425c9f84fe809323e8fe: Deleting consensus metadata
I20260928 15:49:05.881278 30524 catalog_manager.cc:4841] TS 04fad925b878425c9f84fe809323e8fe (127.13.5.66:33725): tablet 97b5359f8dc64ac5a8fffa0022c5a96c (table test-table [id=b49d0604cbcd47119d857836e157a5be]) successfully deleted
I20260928 15:49:05.884507 31047 tablet_replica.cc:330] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: stopping tablet replica
I20260928 15:49:05.885051 31047 raft_consensus.cc:2229] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:05.885651 31047 raft_consensus.cc:2258] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:05.888098 31047 ts_tablet_manager.cc:1905] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:05.888664 31045 tablet_replica.cc:330] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: stopping tablet replica
I20260928 15:49:05.889114 31045 raft_consensus.cc:2229] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:05.889614 31045 raft_consensus.cc:2258] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:05.891654 31045 ts_tablet_manager.cc:1905] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:49:05 dist-test-slave-j968 krb5kdc[30493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:49:05.897629 31047 ts_tablet_manager.cc:1918] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:05.897847 31047 log.cc:1196] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/97b5359f8dc64ac5a8fffa0022c5a96c
I20260928 15:49:05.898409 31047 ts_tablet_manager.cc:1939] T 97b5359f8dc64ac5a8fffa0022c5a96c P 03a59f5dd14a4a07ae93f263e4704a00: Deleting consensus metadata
I20260928 15:49:05.899139 31045 ts_tablet_manager.cc:1918] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:05.899349 31045 log.cc:1196] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/97b5359f8dc64ac5a8fffa0022c5a96c
I20260928 15:49:05.900005 31045 ts_tablet_manager.cc:1939] T 97b5359f8dc64ac5a8fffa0022c5a96c P 625ead7f391a474ca812397093a3d65f: Deleting consensus metadata
I20260928 15:49:05.900765 30525 catalog_manager.cc:4841] TS 03a59f5dd14a4a07ae93f263e4704a00 (127.13.5.65:39455): tablet 97b5359f8dc64ac5a8fffa0022c5a96c (table test-table [id=b49d0604cbcd47119d857836e157a5be]) successfully deleted
I20260928 15:49:05.901865 30522 catalog_manager.cc:4841] TS 625ead7f391a474ca812397093a3d65f (127.13.5.67:42551): tablet 97b5359f8dc64ac5a8fffa0022c5a96c (table test-table [id=b49d0604cbcd47119d857836e157a5be]) successfully deleted
I20260928 15:49:05.904722 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:05.858868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.13.5.65:39455)
0928 15:49:05.859105 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:05.859114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:05.883699 (+ 24585us) server_negotiation.cc:407] Connection header received
0928 15:49:05.884024 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:05.884034 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:05.884187 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:05.884298 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:05.886921 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.888461 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.890154 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:05.890795 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:05.890906 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:05.897021 (+ 6115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:05.897037 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:05.897057 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:05.897105 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:05.901744 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:05.902538 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:05.902549 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:05.902556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:05.902691 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:05.903086 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:05.903092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:05.903096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:05.903447 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:05.903767 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:05.904109 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 15:49:05.904340 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:49:05.905429 30726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:06.005195 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 30599
I20260928 15:49:06.024163 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 30737
Sep 28 15:49:06 dist-test-slave-j968 krb5kdc[30493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:49:06.042520 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:06.025083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.13.5.66:33725)
0928 15:49:06.025263 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:06.025267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:06.025285 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:49:06.025389 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:06.025508 (+ 119us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:06.025582 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:06.025668 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:06.027306 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:06.028233 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:06.030067 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:06.030505 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:06.030600 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:06.036049 (+ 5449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:06.036071 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:06.036094 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:06.036155 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:06.039240 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:06.040213 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:06.040223 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:06.040227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:06.040445 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:06.040857 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:06.040862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:06.040865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:06.041267 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:06.041683 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:06.041895 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 15:49:06.042134 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:49:06.043346 30862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:06.164966 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 30737
I20260928 15:49:06.184970 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 30872
Sep 28 15:49:06 dist-test-slave-j968 krb5kdc[30493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:49:06.202255 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:06.185882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.13.5.67:42551)
0928 15:49:06.186113 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:06.186119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:06.186141 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:49:06.186338 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:06.186347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:06.186434 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:06.186550 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:06.187966 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:06.188705 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:06.190167 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:06.190597 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:06.190742 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:06.195921 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:06.195944 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:06.195966 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:06.196036 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:06.199474 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:06.200317 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:06.200324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:06.200327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:06.200468 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:06.200840 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:06.200844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:06.200846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:06.201220 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:06.201474 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:06.201810 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 15:49:06.201983 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:49:06.203022 30997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:06.293038 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 30872
I20260928 15:49:06.311187 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 30506
I20260928 15:49:06.312510 30568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:06.426692 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 30506
2026-09-28T15:49:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4721 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T15:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:06Z Disabled control of system clock
I20260928 15:49:06.511986 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:39401
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:39401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:49:06.857275 31056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:06.857583 31056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:06.857666 31056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:06.865006 31056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:06.865127 31056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:06.865164 31056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:06.865214 31056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:06.865259 31056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:06.874722 31056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:39401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:39401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:06.876040 31056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:06.877928 31056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:06.890447 31062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:06.890537 31061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:06.890499 31065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:06.891168 31056 server_base.cc:1034] running on GCE node
I20260928 15:49:06.892072 31056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:06.893491 31056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:06.894716 31056 hybrid_clock.cc:648] HybridClock initialized: now 1790610546894649 us; error 51 us; skew 500 ppm
I20260928 15:49:06.895042 31056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:06.909268 31056 webserver.cc:460] Webserver started at http://127.13.5.126:39275/ using document root <none> and password file <none>
I20260928 15:49:06.910056 31056 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:06.910154 31056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:06.910425 31056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:06.913478 31056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "742911ba8ae24538b124deae630b8358"
format_stamp: "Formatted at 2026-09-28 15:49:06 on dist-test-slave-j968"
server_key: "8878fb26e8368f8517a2d18acde9876b"
server_key_iv: "407c925384b6f24a3d9758a10f9b354d"
server_key_version: "encryptionkey@0"
I20260928 15:49:06.914237 31056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "742911ba8ae24538b124deae630b8358"
format_stamp: "Formatted at 2026-09-28 15:49:06 on dist-test-slave-j968"
server_key: "8878fb26e8368f8517a2d18acde9876b"
server_key_iv: "407c925384b6f24a3d9758a10f9b354d"
server_key_version: "encryptionkey@0"
I20260928 15:49:06.919492 31056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:49:06.923102 31072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:06.924502 31056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:06.924688 31056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "742911ba8ae24538b124deae630b8358"
format_stamp: "Formatted at 2026-09-28 15:49:06 on dist-test-slave-j968"
server_key: "8878fb26e8368f8517a2d18acde9876b"
server_key_iv: "407c925384b6f24a3d9758a10f9b354d"
server_key_version: "encryptionkey@0"
I20260928 15:49:06.924868 31056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:06.950743 31056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:06.951632 31056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:06.951917 31056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:06.971436 31056 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:39401
I20260928 15:49:06.971442 31124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:39401 every 8 connection(s)
I20260928 15:49:06.972891 31056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:06.977139 31125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:06.978519 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 31056
I20260928 15:49:06.978711 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:06.979202 13333 external_mini_cluster.cc:1428] Setting key a252d10cc21ca5af3d88fba0e7c3ad41
I20260928 15:49:06.992349 31125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358: Bootstrap starting.
I20260928 15:49:06.993726 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:06.981193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.13.5.126:39401)
0928 15:49:06.982235 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:06.982246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:06.982293 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:49:06.983826 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:06.983848 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:06.984409 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:06.984747 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:06.985460 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:06.986754 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:06.988972 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:06.989794 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:06.989938 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:06.991291 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:06.991298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:06.991312 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:06.991967 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:06.992061 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:06.992084 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:06.992178 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:06.992427 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 15:49:06.992752 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":378,"threads_started":1}
I20260928 15:49:06.995767 31125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:06.997043 31125 log.cc:824] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:07.000065 31125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358: No bootstrap required, opened a new log
I20260928 15:49:07.008014 31125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742911ba8ae24538b124deae630b8358" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39401 } }
I20260928 15:49:07.008384 31125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:07.008461 31125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742911ba8ae24538b124deae630b8358, State: Initialized, Role: FOLLOWER
I20260928 15:49:07.009011 31125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742911ba8ae24538b124deae630b8358" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39401 } }
I20260928 15:49:07.009222 31125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:07.009323 31125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:07.009450 31125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:07.012209 31125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742911ba8ae24538b124deae630b8358" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39401 } }
I20260928 15:49:07.012678 31125 leader_election.cc:302] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742911ba8ae24538b124deae630b8358; no voters:
I20260928 15:49:07.013213 31125 leader_election.cc:288] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:07.013396 31130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:07.014166 31130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [term 1 LEADER]: Becoming Leader. State: Replica: 742911ba8ae24538b124deae630b8358, State: Running, Role: LEADER
I20260928 15:49:07.014778 31130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742911ba8ae24538b124deae630b8358" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39401 } }
I20260928 15:49:07.015460 31125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:07.018018 31132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742911ba8ae24538b124deae630b8358. Latest consensus state: current_term: 1 leader_uuid: "742911ba8ae24538b124deae630b8358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742911ba8ae24538b124deae630b8358" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39401 } } }
I20260928 15:49:07.018553 31132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:07.018218 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742911ba8ae24538b124deae630b8358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742911ba8ae24538b124deae630b8358" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39401 } } }
I20260928 15:49:07.019140 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:07.020406 31139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:07.025751 31139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:07.034626 31139 catalog_manager.cc:1347] Generated new cluster ID: 0a1d57c165cc4ee88070e5f1eaa75acc
I20260928 15:49:07.034737 31139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:07.054127 31139 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:07.055456 31139 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:07.065742 31139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358: Generated new TSK 0
I20260928 15:49:07.066879 31139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:07.084048 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:39401
--builtin_ntp_servers=127.13.5.84:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:49:07.448257 31149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:07.448529 31149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:07.448697 31149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:07.456087 31149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:07.456189 31149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:07.456357 31149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:49:07.465612 31149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:07.467015 31149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:07.468967 31149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:07.482779 31155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:07.482779 31154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:07.482982 31157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:07.483829 31149 server_base.cc:1034] running on GCE node
I20260928 15:49:07.484602 31149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:07.485623 31149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:07.486873 31149 hybrid_clock.cc:648] HybridClock initialized: now 1790610547486834 us; error 36 us; skew 500 ppm
I20260928 15:49:07.487211 31149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:07.489706 31149 webserver.cc:460] Webserver started at http://127.13.5.65:38391/ using document root <none> and password file <none>
I20260928 15:49:07.490484 31149 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:07.490626 31149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:07.490960 31149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:07.494311 31149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "e124660f3dd04235b3b1c7adc5b478c3"
format_stamp: "Formatted at 2026-09-28 15:49:07 on dist-test-slave-j968"
server_key: "f348c3f3fd817ea52a1b82ec034566f5"
server_key_iv: "947de3783d2a409a81eef71696f62349"
server_key_version: "encryptionkey@0"
I20260928 15:49:07.495172 31149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "e124660f3dd04235b3b1c7adc5b478c3"
format_stamp: "Formatted at 2026-09-28 15:49:07 on dist-test-slave-j968"
server_key: "f348c3f3fd817ea52a1b82ec034566f5"
server_key_iv: "947de3783d2a409a81eef71696f62349"
server_key_version: "encryptionkey@0"
I20260928 15:49:07.500945 31149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 15:49:07.505008 31163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:07.506547 31149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:07.506736 31149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "e124660f3dd04235b3b1c7adc5b478c3"
format_stamp: "Formatted at 2026-09-28 15:49:07 on dist-test-slave-j968"
server_key: "f348c3f3fd817ea52a1b82ec034566f5"
server_key_iv: "947de3783d2a409a81eef71696f62349"
server_key_version: "encryptionkey@0"
I20260928 15:49:07.506994 31149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:07.525223 31149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:07.526119 31149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:07.526477 31149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:07.527566 31149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:07.529482 31149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:07.529582 31149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:07.529716 31149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:07.529785 31149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:07.565394 31149 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:37821
I20260928 15:49:07.565443 31278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:37821 every 8 connection(s)
I20260928 15:49:07.566920 31149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:49:07.573598 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 31149
I20260928 15:49:07.573824 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:49:07.574335 13333 external_mini_cluster.cc:1428] Setting key d962e9d9d7ab548f0031a8c6296f4cdf
I20260928 15:49:07.577615 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:39401
--builtin_ntp_servers=127.13.5.84:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:49:07.582233 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:07.569880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:49675 (local address 127.13.5.126:39401)
0928 15:49:07.570362 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:07.570367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:07.571948 (+ 1581us) server_negotiation.cc:407] Connection header received
0928 15:49:07.574146 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:07.574152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:07.574256 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:07.574404 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:07.576008 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:07.576988 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:07.578709 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:07.579429 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:07.579583 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:07.581050 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:07.581059 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:07.581066 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:07.581093 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:07.581144 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:07.581162 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:07.581321 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:07.581646 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 15:49:07.581775 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260928 15:49:07.583164 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:07.570539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39401 (local address 127.13.5.65:49675)
0928 15:49:07.571745 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:07.571787 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:07.573769 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:07.574674 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:07.574684 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:07.574801 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:07.575751 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:07.575780 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:07.577170 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:07.577174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:07.578544 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:07.578557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:07.579661 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:07.579666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:07.579955 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:07.580695 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:07.580697 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:07.580743 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:07.580828 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:07.581371 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:07.581389 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:07.581396 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:49:07.581584 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 15:49:07.581970 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":360,"threads_started":1}
I20260928 15:49:07.584950 31279 heartbeater.cc:347] Connected to a master server at 127.13.5.126:39401
I20260928 15:49:07.585351 31279 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:07.586510 31279 heartbeater.cc:511] Master 127.13.5.126:39401 requested a full tablet report, sending...
I20260928 15:49:07.589440 31090 ts_manager.cc:194] Registered new tserver with Master: e124660f3dd04235b3b1c7adc5b478c3 (127.13.5.65:37821)
I20260928 15:49:07.592005 31090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.65:49675
W20260928 15:49:07.920724 31283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:07.921054 31283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:07.921154 31283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:07.928741 31283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:07.928841 31283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:07.928987 31283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:49:07.938278 31283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:07.939572 31283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:07.941371 31283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:07.955348 31288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:07.955271 31291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:07.955296 31289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:07.956038 31283 server_base.cc:1034] running on GCE node
I20260928 15:49:07.956766 31283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:07.957724 31283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:07.958997 31283 hybrid_clock.cc:648] HybridClock initialized: now 1790610547958899 us; error 58 us; skew 500 ppm
I20260928 15:49:07.959353 31283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:07.961961 31283 webserver.cc:460] Webserver started at http://127.13.5.66:46219/ using document root <none> and password file <none>
I20260928 15:49:07.962688 31283 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:07.962826 31283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:07.963158 31283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:07.966290 31283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "03110cc0355444bd9d6b60db1dcd79bc"
format_stamp: "Formatted at 2026-09-28 15:49:07 on dist-test-slave-j968"
server_key: "f554ae3af2a4121cd435c7031256ad6a"
server_key_iv: "576a9be2ed9b854448281072f2e0c871"
server_key_version: "encryptionkey@0"
I20260928 15:49:07.967113 31283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "03110cc0355444bd9d6b60db1dcd79bc"
format_stamp: "Formatted at 2026-09-28 15:49:07 on dist-test-slave-j968"
server_key: "f554ae3af2a4121cd435c7031256ad6a"
server_key_iv: "576a9be2ed9b854448281072f2e0c871"
server_key_version: "encryptionkey@0"
I20260928 15:49:07.972625 31283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 15:49:07.976351 31297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:07.977921 31283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:49:07.978104 31283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "03110cc0355444bd9d6b60db1dcd79bc"
format_stamp: "Formatted at 2026-09-28 15:49:07 on dist-test-slave-j968"
server_key: "f554ae3af2a4121cd435c7031256ad6a"
server_key_iv: "576a9be2ed9b854448281072f2e0c871"
server_key_version: "encryptionkey@0"
I20260928 15:49:07.978320 31283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:07.998931 31283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:07.999828 31283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:08.000156 31283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:08.001164 31283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:08.002769 31283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:08.002907 31283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:08.003037 31283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:08.003127 31283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:08.038398 31283 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:40731
I20260928 15:49:08.038470 31410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:40731 every 8 connection(s)
I20260928 15:49:08.039712 31283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:49:08.048744 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 31283
I20260928 15:49:08.048943 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:49:08.049474 13333 external_mini_cluster.cc:1428] Setting key df7e8410d88e3836fe1fed29387c8740
I20260928 15:49:08.052591 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:39401
--builtin_ntp_servers=127.13.5.84:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:49:08.054277 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.042622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:35477 (local address 127.13.5.126:39401)
0928 15:49:08.042837 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.042841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.044473 (+ 1632us) server_negotiation.cc:407] Connection header received
0928 15:49:08.046478 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.046484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.046586 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.046719 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:08.047886 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.049036 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.050875 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.051515 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.051670 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.053361 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:08.053366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:08.053369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:08.053387 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:08.053419 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:08.053431 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:08.053621 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.053853 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.053962 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 15:49:08.055392 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.043235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39401 (local address 127.13.5.66:35477)
0928 15:49:08.044298 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:08.044337 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:08.046167 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:08.046970 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:08.046979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:08.047074 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:08.047679 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.047698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.049213 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.049218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.050729 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.050742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.051777 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.051786 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.052126 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.053005 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:08.053008 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:08.053054 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:08.053147 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:08.053618 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:08.053629 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:08.053640 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:49:08.053842 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 15:49:08.054290 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":308,"threads_started":1}
I20260928 15:49:08.056999 31411 heartbeater.cc:347] Connected to a master server at 127.13.5.126:39401
I20260928 15:49:08.057348 31411 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:08.058173 31411 heartbeater.cc:511] Master 127.13.5.126:39401 requested a full tablet report, sending...
I20260928 15:49:08.059727 31090 ts_manager.cc:194] Registered new tserver with Master: 03110cc0355444bd9d6b60db1dcd79bc (127.13.5.66:40731)
I20260928 15:49:08.061044 31090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.66:35477
W20260928 15:49:08.396330 31414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:08.396646 31414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:08.396803 31414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:08.404285 31414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:08.404398 31414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:08.404569 31414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:49:08.414140 31414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:08.415581 31414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:08.417550 31414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:08.431311 31419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:08.431319 31420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:08.431435 31422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:08.432331 31414 server_base.cc:1034] running on GCE node
I20260928 15:49:08.432906 31414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:08.433867 31414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:08.435098 31414 hybrid_clock.cc:648] HybridClock initialized: now 1790610548435035 us; error 33 us; skew 500 ppm
I20260928 15:49:08.435395 31414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:08.437733 31414 webserver.cc:460] Webserver started at http://127.13.5.67:37677/ using document root <none> and password file <none>
I20260928 15:49:08.438424 31414 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:08.438513 31414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:08.438768 31414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:08.441867 31414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "df60f5477ba24125b318019f9e85b0da"
format_stamp: "Formatted at 2026-09-28 15:49:08 on dist-test-slave-j968"
server_key: "b2abb34851df849ca89838aba140cf81"
server_key_iv: "1e697d22afaaeb89b70212556801518b"
server_key_version: "encryptionkey@0"
I20260928 15:49:08.442662 31414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "df60f5477ba24125b318019f9e85b0da"
format_stamp: "Formatted at 2026-09-28 15:49:08 on dist-test-slave-j968"
server_key: "b2abb34851df849ca89838aba140cf81"
server_key_iv: "1e697d22afaaeb89b70212556801518b"
server_key_version: "encryptionkey@0"
I20260928 15:49:08.448104 31414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260928 15:49:08.451714 31428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:08.453158 31414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:49:08.453298 31414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "df60f5477ba24125b318019f9e85b0da"
format_stamp: "Formatted at 2026-09-28 15:49:08 on dist-test-slave-j968"
server_key: "b2abb34851df849ca89838aba140cf81"
server_key_iv: "1e697d22afaaeb89b70212556801518b"
server_key_version: "encryptionkey@0"
I20260928 15:49:08.453481 31414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:08.468644 31414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:08.469470 31414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:08.469760 31414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:08.470692 31414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:08.472203 31414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:08.472290 31414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:08.472373 31414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:08.472427 31414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:08.507318 31414 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:44085
I20260928 15:49:08.507371 31544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:44085 every 8 connection(s)
I20260928 15:49:08.508786 31414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:49:08.510986 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 31414
I20260928 15:49:08.511183 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:49:08.511817 13333 external_mini_cluster.cc:1428] Setting key 988199627bf5aeb682b212818b6ae5ab
I20260928 15:49:08.523192 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.511743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:49377 (local address 127.13.5.126:39401)
0928 15:49:08.511977 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.511983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.513932 (+ 1949us) server_negotiation.cc:407] Connection header received
0928 15:49:08.516206 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.516210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.516290 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.516372 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:08.517538 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.518303 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.520522 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.520944 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.521040 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.522455 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:08.522460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:08.522463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:08.522484 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:08.522517 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:08.522528 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:08.522623 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.522851 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.522940 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:49:08.524467 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.512469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39401 (local address 127.13.5.67:49377)
0928 15:49:08.513736 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:08.513784 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:08.515899 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:08.516619 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:08.516627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:08.516720 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:08.517377 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.517395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.518595 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.518600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.520360 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.520374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.521169 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.521174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.521406 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.522131 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:08.522135 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:08.522178 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:08.522262 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:08.522649 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:08.522657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:08.522664 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:49:08.522827 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 15:49:08.523201 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":346,"threads_started":1}
I20260928 15:49:08.526135 31545 heartbeater.cc:347] Connected to a master server at 127.13.5.126:39401
I20260928 15:49:08.526479 31545 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:08.527590 31545 heartbeater.cc:511] Master 127.13.5.126:39401 requested a full tablet report, sending...
I20260928 15:49:08.529394 31090 ts_manager.cc:194] Registered new tserver with Master: df60f5477ba24125b318019f9e85b0da (127.13.5.67:44085)
I20260928 15:49:08.530347 31090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.67:49377
I20260928 15:49:08.540263 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:49:08.550369 31281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.544575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.13.5.126:39401)
0928 15:49:08.544783 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.544787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.545182 (+ 395us) server_negotiation.cc:407] Connection header received
0928 15:49:08.545324 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.545328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.545417 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.545508 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:08.545986 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.546674 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.548048 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.548471 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.548580 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.549626 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:08.549631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:08.549634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:08.549653 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:08.549688 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:08.549700 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:08.549773 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.549942 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.550061 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:49:08.554195 31090 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:49:08.561245 31090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:49:08.595932 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.577323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.13.5.66:40731)
0928 15:49:08.587525 (+ 10202us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.587538 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.588860 (+ 1322us) server_negotiation.cc:407] Connection header received
0928 15:49:08.588997 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.589005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.589228 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.589431 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:08.590032 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.591404 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.593635 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.594703 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.594877 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.595142 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.595382 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.595559 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10027,"thread_start_us":9928,"threads_started":1}
I20260928 15:49:08.595979 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.576745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:40731 (local address 127.0.0.1:60712)
0928 15:49:08.588729 (+ 11984us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:08.588745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:08.588858 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:08.589439 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:08.589445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:08.589470 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:08.589852 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.589863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.591594 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.591599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.593408 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.593423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.594907 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.594913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.595123 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.595182 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 15:49:08.595342 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 15:49:08.595544 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11869,"mutex_wait_us":27,"thread_start_us":10963,"threads_started":1}
I20260928 15:49:08.596405 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.574236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:44085 (local address 127.0.0.1:41358)
0928 15:49:08.588042 (+ 13806us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:08.588064 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:08.588205 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:08.589103 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:08.589110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:08.589138 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:08.589470 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.589482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.591400 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.591406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.593772 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.593787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.595596 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.595602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.595831 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.595859 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 15:49:08.595957 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:49:08.596075 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13648,"mutex_wait_us":23,"thread_start_us":9875,"threads_started":1}
I20260928 15:49:08.596637 31279 heartbeater.cc:503] Master 127.13.5.126:39401 was elected leader, sending a full tablet report...
I20260928 15:49:08.597922 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.574217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.13.5.65:37821)
0928 15:49:08.587104 (+ 12887us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.587118 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.587159 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:49:08.587302 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.587315 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.587681 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.588727 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:08.589008 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.590469 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.593434 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.594836 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.594974 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.595249 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.596110 (+ 861us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.596276 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12649,"thread_start_us":12527,"threads_started":1}
I20260928 15:49:08.597911 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.575129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.13.5.67:44085)
0928 15:49:08.588453 (+ 13324us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.588462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.588521 (+ 59us) server_negotiation.cc:407] Connection header received
0928 15:49:08.588622 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.588629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.588885 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.589688 (+ 803us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:08.589964 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.591242 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.593967 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.595371 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.596972 (+ 1601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.597284 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.597354 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.597515 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13119,"thread_start_us":12958,"threads_started":1}
I20260928 15:49:08.598711 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.573910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:37821 (local address 127.0.0.1:53494)
0928 15:49:08.584634 (+ 10724us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:08.584705 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:08.584929 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:08.588007 (+ 3078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:08.588023 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:08.588065 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:08.588538 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.588555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.590635 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.590640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.593206 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.593224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.595394 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.595400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.595679 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.595723 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 15:49:08.595795 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:49:08.595964 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10340,"mutex_wait_us":33,"thread_start_us":15043,"threads_started":1}
I20260928 15:49:08.600453 31346 tablet_service.cc:1467] Processing CreateTablet for tablet 04da197ba5a2401fb523c9416ed87f21 (DEFAULT_TABLE table=test-table [id=7fd4fdbcf8614abb850e5eb6d6308285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:08.601742 31346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04da197ba5a2401fb523c9416ed87f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:08.601606 31478 tablet_service.cc:1467] Processing CreateTablet for tablet 04da197ba5a2401fb523c9416ed87f21 (DEFAULT_TABLE table=test-table [id=7fd4fdbcf8614abb850e5eb6d6308285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:08.602231 31212 tablet_service.cc:1467] Processing CreateTablet for tablet 04da197ba5a2401fb523c9416ed87f21 (DEFAULT_TABLE table=test-table [id=7fd4fdbcf8614abb850e5eb6d6308285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:08.603116 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04da197ba5a2401fb523c9416ed87f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:08.603615 31212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04da197ba5a2401fb523c9416ed87f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:08.614902 31563 tablet_bootstrap.cc:492] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Bootstrap starting.
I20260928 15:49:08.617878 31564 tablet_bootstrap.cc:492] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Bootstrap starting.
I20260928 15:49:08.618675 31565 tablet_bootstrap.cc:492] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Bootstrap starting.
I20260928 15:49:08.618672 31563 tablet_bootstrap.cc:654] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:08.620129 31563 log.cc:824] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:08.622210 31565 tablet_bootstrap.cc:654] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:08.622923 31564 tablet_bootstrap.cc:654] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:08.623446 31565 log.cc:824] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:08.623813 31563 tablet_bootstrap.cc:492] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: No bootstrap required, opened a new log
I20260928 15:49:08.623982 31564 log.cc:824] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:08.624294 31563 ts_tablet_manager.cc:1397] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260928 15:49:08.626690 31564 tablet_bootstrap.cc:492] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: No bootstrap required, opened a new log
I20260928 15:49:08.626956 31564 ts_tablet_manager.cc:1397] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 15:49:08.630641 31565 tablet_bootstrap.cc:492] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: No bootstrap required, opened a new log
I20260928 15:49:08.630942 31565 ts_tablet_manager.cc:1397] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 15:49:08.634648 31564 raft_consensus.cc:348] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.635046 31564 raft_consensus.cc:374] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:08.635208 31564 raft_consensus.cc:729] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e124660f3dd04235b3b1c7adc5b478c3, State: Initialized, Role: FOLLOWER
I20260928 15:49:08.635896 31564 consensus_queue.cc:260] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.636155 31563 raft_consensus.cc:348] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.636650 31563 raft_consensus.cc:374] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:08.637552 31564 ts_tablet_manager.cc:1428] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260928 15:49:08.636780 31563 raft_consensus.cc:729] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03110cc0355444bd9d6b60db1dcd79bc, State: Initialized, Role: FOLLOWER
I20260928 15:49:08.638484 31563 consensus_queue.cc:260] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.639983 31411 heartbeater.cc:503] Master 127.13.5.126:39401 was elected leader, sending a full tablet report...
I20260928 15:49:08.640635 31563 ts_tablet_manager.cc:1428] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260928 15:49:08.641364 31565 raft_consensus.cc:348] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.641912 31565 raft_consensus.cc:374] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:08.642038 31565 raft_consensus.cc:729] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df60f5477ba24125b318019f9e85b0da, State: Initialized, Role: FOLLOWER
I20260928 15:49:08.642684 31565 consensus_queue.cc:260] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.644586 31545 heartbeater.cc:503] Master 127.13.5.126:39401 was elected leader, sending a full tablet report...
I20260928 15:49:08.644975 31565 ts_tablet_manager.cc:1428] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260928 15:49:08.760267 31569 raft_consensus.cc:482] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260928 15:49:08.760674 31546 tablet.cc:2369] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:49:08.760643 31569 raft_consensus.cc:504] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.762655 31569 leader_election.cc:288] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df60f5477ba24125b318019f9e85b0da (127.13.5.67:44085), 03110cc0355444bd9d6b60db1dcd79bc (127.13.5.66:40731)
I20260928 15:49:08.770496 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.763209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:41873 (local address 127.13.5.67:44085)
0928 15:49:08.763416 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.763424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.764544 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 15:49:08.764744 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.764749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.764820 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.764919 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:08.765566 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.766540 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.768682 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.769480 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.769637 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.769753 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.770081 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.770220 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:49:08.770514 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.763275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:50527 (local address 127.13.5.66:40731)
0928 15:49:08.763503 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.763509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.764486 (+ 977us) server_negotiation.cc:407] Connection header received
0928 15:49:08.764722 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.764726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.764798 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.764902 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:08.765513 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.766311 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.768712 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.769470 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.769633 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.769735 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.770080 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.770235 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:49:08.770637 31572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.763217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:40731 (local address 127.13.5.65:50527)
0928 15:49:08.764367 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:08.764383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:08.764558 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:08.764932 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:08.764936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:08.764961 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:08.765360 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.765370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.766443 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.766448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.768480 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.768493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.769690 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.769697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.769885 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.769939 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 15:49:08.770048 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:49:08.770250 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":201,"thread_start_us":685,"threads_started":1}
I20260928 15:49:08.770676 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.763216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:44085 (local address 127.13.5.65:41873)
0928 15:49:08.764431 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:08.764450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:08.764613 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:08.764945 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:08.764949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:08.764972 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:08.765393 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.765401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.766663 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.766669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.768479 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:08.768493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.769727 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.769735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:08.769896 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.769945 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 15:49:08.770051 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:49:08.770250 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":148,"thread_start_us":626,"threads_started":1}
I20260928 15:49:08.771795 31366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04da197ba5a2401fb523c9416ed87f21" candidate_uuid: "e124660f3dd04235b3b1c7adc5b478c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03110cc0355444bd9d6b60db1dcd79bc" is_pre_election: true
I20260928 15:49:08.772341 31498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04da197ba5a2401fb523c9416ed87f21" candidate_uuid: "e124660f3dd04235b3b1c7adc5b478c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df60f5477ba24125b318019f9e85b0da" is_pre_election: true
I20260928 15:49:08.772501 31366 raft_consensus.cc:2454] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e124660f3dd04235b3b1c7adc5b478c3 in term 0.
I20260928 15:49:08.772898 31498 raft_consensus.cc:2454] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e124660f3dd04235b3b1c7adc5b478c3 in term 0.
I20260928 15:49:08.773296 31164 leader_election.cc:302] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03110cc0355444bd9d6b60db1dcd79bc, e124660f3dd04235b3b1c7adc5b478c3; no voters:
I20260928 15:49:08.773906 31569 raft_consensus.cc:2781] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:49:08.774057 31569 raft_consensus.cc:482] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:49:08.774147 31569 raft_consensus.cc:3037] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:08.776485 31569 raft_consensus.cc:504] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.777164 31569 leader_election.cc:288] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [CANDIDATE]: Term 1 election: Requested vote from peers df60f5477ba24125b318019f9e85b0da (127.13.5.67:44085), 03110cc0355444bd9d6b60db1dcd79bc (127.13.5.66:40731)
I20260928 15:49:08.777654 31498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04da197ba5a2401fb523c9416ed87f21" candidate_uuid: "e124660f3dd04235b3b1c7adc5b478c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df60f5477ba24125b318019f9e85b0da"
I20260928 15:49:08.777760 31366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04da197ba5a2401fb523c9416ed87f21" candidate_uuid: "e124660f3dd04235b3b1c7adc5b478c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03110cc0355444bd9d6b60db1dcd79bc"
I20260928 15:49:08.777868 31498 raft_consensus.cc:3037] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:08.778026 31366 raft_consensus.cc:3037] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:08.780223 31498 raft_consensus.cc:2454] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e124660f3dd04235b3b1c7adc5b478c3 in term 1.
I20260928 15:49:08.780455 31366 raft_consensus.cc:2454] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e124660f3dd04235b3b1c7adc5b478c3 in term 1.
I20260928 15:49:08.780752 31165 leader_election.cc:302] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df60f5477ba24125b318019f9e85b0da, e124660f3dd04235b3b1c7adc5b478c3; no voters:
I20260928 15:49:08.781072 31569 raft_consensus.cc:2781] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:08.781603 31569 raft_consensus.cc:686] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 1 LEADER]: Becoming Leader. State: Replica: e124660f3dd04235b3b1c7adc5b478c3, State: Running, Role: LEADER
I20260928 15:49:08.782204 31569 consensus_queue.cc:237] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } }
I20260928 15:49:08.786964 31089 catalog_manager.cc:5495] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e124660f3dd04235b3b1c7adc5b478c3 (127.13.5.65). New cstate: current_term: 1 leader_uuid: "e124660f3dd04235b3b1c7adc5b478c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e124660f3dd04235b3b1c7adc5b478c3" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 37821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 } health_report { overall_health: UNKNOWN } } }
W20260928 15:49:08.791945 31412 tablet.cc:2369] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:49:08.819633 31280 tablet.cc:2369] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:49:08.825978 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.820753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.13.5.65:37821)
0928 15:49:08.820922 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.820926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.820946 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:49:08.821058 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.821062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.821148 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.821279 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:49:08.821662 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.822362 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.823743 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.824251 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.824352 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.824838 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:49:08.825348 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:49:08.825526 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.825614 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.825706 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:49:08.834827 31498 raft_consensus.cc:1261] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 1 FOLLOWER]: Refusing update from remote peer e124660f3dd04235b3b1c7adc5b478c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:08.834825 31366 raft_consensus.cc:1261] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 1 FOLLOWER]: Refusing update from remote peer e124660f3dd04235b3b1c7adc5b478c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:08.835855 31574 consensus_queue.cc:1035] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df60f5477ba24125b318019f9e85b0da" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:08.836292 31569 consensus_queue.cc:1035] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03110cc0355444bd9d6b60db1dcd79bc" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:08.856537 31579 mvcc.cc:205] Tried to move back new op lower bound from 7334340808016535552 to 7334340807816634368. Current Snapshot: MvccSnapshot[applied={T|T < 7334340808016535552}]
I20260928 15:49:08.865525 31090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:08.865857 31090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:08.869984 31090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 742911ba8ae24538b124deae630b8358: Sending DeleteTablet for 3 replicas of tablet 04da197ba5a2401fb523c9416ed87f21
I20260928 15:49:08.871301 31212 tablet_service.cc:1514] Processing DeleteTablet for tablet 04da197ba5a2401fb523c9416ed87f21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:08 UTC) from {username='slave'} at 127.0.0.1:53494
I20260928 15:49:08.871301 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet 04da197ba5a2401fb523c9416ed87f21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:08 UTC) from {username='slave'} at 127.0.0.1:41358
I20260928 15:49:08.871305 31346 tablet_service.cc:1514] Processing DeleteTablet for tablet 04da197ba5a2401fb523c9416ed87f21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:08 UTC) from {username='slave'} at 127.0.0.1:60712
I20260928 15:49:08.874853 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 31149
I20260928 15:49:08.884020 31589 tablet_replica.cc:330] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: stopping tablet replica
I20260928 15:49:08.884850 31587 tablet_replica.cc:330] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: stopping tablet replica
I20260928 15:49:08.884951 31589 raft_consensus.cc:2229] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:08.885277 31587 raft_consensus.cc:2229] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:08.885368 31588 tablet_replica.cc:330] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: stopping tablet replica
I20260928 15:49:08.885527 31589 raft_consensus.cc:2258] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:08.885784 31587 raft_consensus.cc:2258] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:08.885989 31588 raft_consensus.cc:2229] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:49:08.886911 31588 raft_consensus.cc:2258] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:08.887890 31589 ts_tablet_manager.cc:1905] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:08.889613 31588 ts_tablet_manager.cc:1905] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:08.890883 31587 ts_tablet_manager.cc:1905] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:08.895013 31589 ts_tablet_manager.cc:1918] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:08.895188 31589 log.cc:1196] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/04da197ba5a2401fb523c9416ed87f21
I20260928 15:49:08.895752 31589 ts_tablet_manager.cc:1939] T 04da197ba5a2401fb523c9416ed87f21 P 03110cc0355444bd9d6b60db1dcd79bc: Deleting consensus metadata
I20260928 15:49:08.897716 31073 catalog_manager.cc:4841] TS 03110cc0355444bd9d6b60db1dcd79bc (127.13.5.66:40731): tablet 04da197ba5a2401fb523c9416ed87f21 (table test-table [id=7fd4fdbcf8614abb850e5eb6d6308285]) successfully deleted
I20260928 15:49:08.898705 31588 ts_tablet_manager.cc:1918] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:08.898888 31588 log.cc:1196] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/04da197ba5a2401fb523c9416ed87f21
I20260928 15:49:08.899312 31587 ts_tablet_manager.cc:1918] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:08.899389 31588 ts_tablet_manager.cc:1939] T 04da197ba5a2401fb523c9416ed87f21 P e124660f3dd04235b3b1c7adc5b478c3: Deleting consensus metadata
I20260928 15:49:08.899482 31587 log.cc:1196] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/04da197ba5a2401fb523c9416ed87f21
I20260928 15:49:08.900067 31587 ts_tablet_manager.cc:1939] T 04da197ba5a2401fb523c9416ed87f21 P df60f5477ba24125b318019f9e85b0da: Deleting consensus metadata
I20260928 15:49:08.901234 31076 catalog_manager.cc:4841] TS e124660f3dd04235b3b1c7adc5b478c3 (127.13.5.65:37821): tablet 04da197ba5a2401fb523c9416ed87f21 (table test-table [id=7fd4fdbcf8614abb850e5eb6d6308285]) successfully deleted
I20260928 15:49:08.902369 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:08.876074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.13.5.65:37821)
0928 15:49:08.876263 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:08.876269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:08.886240 (+ 9971us) server_negotiation.cc:407] Connection header received
0928 15:49:08.886535 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:08.886541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:08.886685 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:08.886770 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:08.887475 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.888717 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.890721 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:08.891405 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:08.891522 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:08.900571 (+ 9049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:08.900583 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:08.900600 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:08.901409 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:08.901489 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:08.901516 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:08.901634 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:08.901828 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 15:49:08.901950 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:49:08.902407 31074 catalog_manager.cc:4841] TS df60f5477ba24125b318019f9e85b0da (127.13.5.67:44085): tablet 04da197ba5a2401fb523c9416ed87f21 (table test-table [id=7fd4fdbcf8614abb850e5eb6d6308285]) successfully deleted
I20260928 15:49:08.903192 31274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:09.004900 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 31149
I20260928 15:49:09.024778 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 31283
I20260928 15:49:09.032208 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:09.025565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.13.5.66:40731)
0928 15:49:09.025745 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:09.025751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:09.025771 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:49:09.025970 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:09.025976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:09.026075 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:09.026197 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:09.026662 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:09.027495 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:09.028957 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:09.029462 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:09.029600 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:09.030706 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:09.030715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:09.030730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:09.031317 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:09.031366 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:09.031386 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:09.031560 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:09.031682 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 15:49:09.031793 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:49:09.032893 31406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:09.125166 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 31283
I20260928 15:49:09.143620 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 31414
I20260928 15:49:09.150772 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:09.144549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.13.5.67:44085)
0928 15:49:09.144724 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:09.144728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:09.144769 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:49:09.144830 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:09.144834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:09.144918 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:09.145008 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:09.145437 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:09.146298 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:09.147700 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:09.148242 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:09.148343 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:09.149418 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:09.149431 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:09.149451 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:09.150073 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:09.150126 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:09.150147 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:09.150247 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:09.150397 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 15:49:09.150509 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:49:09.151425 31539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:09.242853 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 31414
I20260928 15:49:09.260840 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 31056
I20260928 15:49:09.262032 31120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:09.375649 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 31056
2026-09-28T15:49:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:09 dist-test-slave-j968 krb5kdc[31597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:09 dist-test-slave-j968 krb5kdc[31597](info): set up 2 sockets
Sep 28 15:49:09 dist-test-slave-j968 krb5kdc[31597](info): commencing operation
krb5kdc: starting...
W20260928 15:49:10.510429 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:10 dist-test-slave-j968 krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:10.772929 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:41649
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:41649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:11.113821 31610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:11.114174 31610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:11.114341 31610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:11.122008 31610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:11.122150 31610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:11.122262 31610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:11.122354 31610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:11.122435 31610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:11.132742 31610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:41649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:11.134222 31610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:11.135622 31610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:11.148345 31619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:11.148309 31616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:11.148380 31617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:11.149133 31610 server_base.cc:1034] running on GCE node
I20260928 15:49:11.149933 31610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:11.151473 31610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:11.152707 31610 hybrid_clock.cc:648] HybridClock initialized: now 1790610551152655 us; error 43 us; skew 500 ppm
I20260928 15:49:11.153064 31610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:11 dist-test-slave-j968 krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:11.156256 31610 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:49:11.157819 31610 webserver.cc:460] Webserver started at http://127.13.5.126:44999/ using document root <none> and password file <none>
I20260928 15:49:11.158473 31610 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:11.158572 31610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:11.158814 31610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:11.161859 31610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "1e7b2245cbd844f5a5cab0240fa34e12"
format_stamp: "Formatted at 2026-09-28 15:49:11 on dist-test-slave-j968"
server_key: "0d8eaba44c60656058ce4ea455a80892"
server_key_iv: "80491a065270196337f33106a57e1ff3"
server_key_version: "encryptionkey@0"
I20260928 15:49:11.162580 31610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "1e7b2245cbd844f5a5cab0240fa34e12"
format_stamp: "Formatted at 2026-09-28 15:49:11 on dist-test-slave-j968"
server_key: "0d8eaba44c60656058ce4ea455a80892"
server_key_iv: "80491a065270196337f33106a57e1ff3"
server_key_version: "encryptionkey@0"
I20260928 15:49:11.167856 31610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:49:11.171386 31627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:11.172942 31610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:11.173102 31610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "1e7b2245cbd844f5a5cab0240fa34e12"
format_stamp: "Formatted at 2026-09-28 15:49:11 on dist-test-slave-j968"
server_key: "0d8eaba44c60656058ce4ea455a80892"
server_key_iv: "80491a065270196337f33106a57e1ff3"
server_key_version: "encryptionkey@0"
I20260928 15:49:11.173273 31610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:11.190310 31610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:11.194829 31610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:11.195081 31610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:11.215237 31610 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:41649
I20260928 15:49:11.215261 31679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:41649 every 8 connection(s)
I20260928 15:49:11.216761 31610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:11.218657 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 31610
I20260928 15:49:11.218850 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:11.219416 13333 external_mini_cluster.cc:1428] Setting key 27a4818e664a4f4a72e4648e7f8222b8
I20260928 15:49:11.221642 31680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:49:11 dist-test-slave-j968 krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:11.234135 31680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12: Bootstrap starting.
I20260928 15:49:11.237756 31680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:11.238960 31680 log.cc:824] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:11.242257 31680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12: No bootstrap required, opened a new log
I20260928 15:49:11.242825 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:11.221457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.13.5.126:41649)
0928 15:49:11.222294 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:11.222307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:11.222348 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:49:11.224255 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:11.224279 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:11.224684 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:11.225098 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:11.226723 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:11.228194 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:11.229574 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:11.230248 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:11.230460 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:11.235165 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:11.235173 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:11.235189 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:11.235229 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:11.238473 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:11.239246 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:11.239253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:11.239259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:11.239426 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:11.239855 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:11.239861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:11.239886 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:11.240478 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:11.240800 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:11.241271 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 15:49:11.241733 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":325,"threads_started":1}
I20260928 15:49:11.250994 31680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7b2245cbd844f5a5cab0240fa34e12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41649 } }
I20260928 15:49:11.251513 31680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:11.251637 31680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7b2245cbd844f5a5cab0240fa34e12, State: Initialized, Role: FOLLOWER
I20260928 15:49:11.252334 31680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7b2245cbd844f5a5cab0240fa34e12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41649 } }
I20260928 15:49:11.252591 31680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:11.252755 31680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:11.252969 31680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:11.255489 31680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7b2245cbd844f5a5cab0240fa34e12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41649 } }
I20260928 15:49:11.256125 31680 leader_election.cc:302] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e7b2245cbd844f5a5cab0240fa34e12; no voters:
I20260928 15:49:11.256819 31680 leader_election.cc:288] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:11.256946 31685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:11.258162 31685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [term 1 LEADER]: Becoming Leader. State: Replica: 1e7b2245cbd844f5a5cab0240fa34e12, State: Running, Role: LEADER
I20260928 15:49:11.258694 31685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7b2245cbd844f5a5cab0240fa34e12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41649 } }
I20260928 15:49:11.259225 31680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:11.261451 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e7b2245cbd844f5a5cab0240fa34e12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7b2245cbd844f5a5cab0240fa34e12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41649 } } }
I20260928 15:49:11.261886 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:11.261811 31687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e7b2245cbd844f5a5cab0240fa34e12. Latest consensus state: current_term: 1 leader_uuid: "1e7b2245cbd844f5a5cab0240fa34e12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7b2245cbd844f5a5cab0240fa34e12" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41649 } } }
I20260928 15:49:11.262107 31687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:11.263276 31692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:11.268630 31692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:11.278659 31692 catalog_manager.cc:1347] Generated new cluster ID: 8a64351daeef45bbb785fd9a998bc95e
I20260928 15:49:11.278823 31692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:11.304119 31692 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:11.305599 31692 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:11.313225 31692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12: Generated new TSK 0
I20260928 15:49:11.314373 31692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:49:11.391947 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41649
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:11.740362 31706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:11.740669 31706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:11.740831 31706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:11.749115 31706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:11.749233 31706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:11.749449 31706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:49:11.759575 31706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:11.761270 31706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:11.762776 31706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:11.776594 31714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:11.776614 31712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:11.776760 31711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:11.781679 31706 server_base.cc:1034] running on GCE node
I20260928 15:49:11.782377 31706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:11.783510 31706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:11.784732 31706 hybrid_clock.cc:648] HybridClock initialized: now 1790610551784671 us; error 40 us; skew 500 ppm
I20260928 15:49:11.785171 31706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:11 dist-test-slave-j968 krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:11.788801 31706 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:49:11.790578 31706 webserver.cc:460] Webserver started at http://127.13.5.65:37171/ using document root <none> and password file <none>
I20260928 15:49:11.791350 31706 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:11.791499 31706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:11.791852 31706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:11.795113 31706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "00124c84ea0e4fc4ba042f879d3385a8"
format_stamp: "Formatted at 2026-09-28 15:49:11 on dist-test-slave-j968"
server_key: "222bf78d86898bff4c6147e0dd92238c"
server_key_iv: "678cb785541d668923da1f7c0c67984d"
server_key_version: "encryptionkey@0"
I20260928 15:49:11.796085 31706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "00124c84ea0e4fc4ba042f879d3385a8"
format_stamp: "Formatted at 2026-09-28 15:49:11 on dist-test-slave-j968"
server_key: "222bf78d86898bff4c6147e0dd92238c"
server_key_iv: "678cb785541d668923da1f7c0c67984d"
server_key_version: "encryptionkey@0"
I20260928 15:49:11.801846 31706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:49:11.805980 31721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:11.807443 31706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:49:11.807647 31706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "00124c84ea0e4fc4ba042f879d3385a8"
format_stamp: "Formatted at 2026-09-28 15:49:11 on dist-test-slave-j968"
server_key: "222bf78d86898bff4c6147e0dd92238c"
server_key_iv: "678cb785541d668923da1f7c0c67984d"
server_key_version: "encryptionkey@0"
I20260928 15:49:11.807912 31706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:11.820264 31706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:11.825103 31706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:11.825407 31706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:11.826406 31706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:11.827827 31706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:11.827968 31706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:49:11.828127 31706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:11.828199 31706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:11.862425 31706 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43323
I20260928 15:49:11.862473 31835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43323 every 8 connection(s)
I20260928 15:49:11.863754 31706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:49:11.868426 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 31706
I20260928 15:49:11.868690 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:49:11.869240 13333 external_mini_cluster.cc:1428] Setting key 0801dda7aca3a1d5664b6dcaf7b809a6
Sep 28 15:49:11 dist-test-slave-j968 krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:11.888016 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:11.866535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:49887 (local address 127.13.5.126:41649)
0928 15:49:11.866983 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:11.866988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:11.868140 (+ 1152us) server_negotiation.cc:407] Connection header received
0928 15:49:11.870534 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:11.870540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:11.870628 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:11.870747 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:11.872841 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:11.873530 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:11.876047 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:11.876571 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:11.876676 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:11.880954 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:11.880963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:11.880969 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:11.881017 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:11.884628 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:11.885376 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:11.885385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:11.885392 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:11.885561 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:11.885966 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:11.885974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:11.885980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:11.886351 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:11.886513 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:11.887231 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 15:49:11.887497 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":206,"threads_started":1}
I20260928 15:49:11.888859 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:11.867133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:41649 (local address 127.13.5.65:49887)
0928 15:49:11.867856 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:11.867963 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:11.870246 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:11.870997 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:11.871006 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:11.871741 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:11.872654 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:11.872683 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:11.873702 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:11.873710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:11.875851 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:11.875895 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:11.876815 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:11.876825 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:11.877200 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:11.877947 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:11.877981 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:11.880687 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:11.884776 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:11.884782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:11.884795 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:11.885234 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:11.885667 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:11.885670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:11.885672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:11.885845 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:11.886469 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:11.886516 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:11.886837 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 15:49:11.887059 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 15:49:11.887452 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":293,"threads_started":1}
I20260928 15:49:11.890512 31836 heartbeater.cc:347] Connected to a master server at 127.13.5.126:41649
I20260928 15:49:11.890847 31836 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:11.891748 31836 heartbeater.cc:511] Master 127.13.5.126:41649 requested a full tablet report, sending...
I20260928 15:49:11.894344 31644 ts_manager.cc:194] Registered new tserver with Master: 00124c84ea0e4fc4ba042f879d3385a8 (127.13.5.65:43323)
I20260928 15:49:11.896626 31644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:49887
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:49:11.932385 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41649
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:12.276677 31842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:12.276981 31842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:12.277148 31842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:12.284722 31842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:12.284842 31842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:12.285066 31842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:49:12.295009 31842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:12.296428 31842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:12.297770 31842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:12.311790 31850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:12.311883 31848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:12.311757 31847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:12.312732 31842 server_base.cc:1034] running on GCE node
I20260928 15:49:12.313441 31842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:12.314554 31842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:12.315788 31842 hybrid_clock.cc:648] HybridClock initialized: now 1790610552315733 us; error 49 us; skew 500 ppm
I20260928 15:49:12.316166 31842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:12 dist-test-slave-j968 krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610552, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:12.325114 31842 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:49:12.326886 31842 webserver.cc:460] Webserver started at http://127.13.5.66:40665/ using document root <none> and password file <none>
I20260928 15:49:12.327647 31842 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:12.327802 31842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:12.328181 31842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:12.331212 31842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "8273f12559444f75a1b3555dfd5533c9"
format_stamp: "Formatted at 2026-09-28 15:49:12 on dist-test-slave-j968"
server_key: "2c224a48b2d411d7c25d463a89ce25c7"
server_key_iv: "f5503b0ea6a33a3008ddc728fe849bbf"
server_key_version: "encryptionkey@0"
I20260928 15:49:12.332108 31842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "8273f12559444f75a1b3555dfd5533c9"
format_stamp: "Formatted at 2026-09-28 15:49:12 on dist-test-slave-j968"
server_key: "2c224a48b2d411d7c25d463a89ce25c7"
server_key_iv: "f5503b0ea6a33a3008ddc728fe849bbf"
server_key_version: "encryptionkey@0"
I20260928 15:49:12.337738 31842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 15:49:12.341674 31857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:12.343197 31842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:49:12.343390 31842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "8273f12559444f75a1b3555dfd5533c9"
format_stamp: "Formatted at 2026-09-28 15:49:12 on dist-test-slave-j968"
server_key: "2c224a48b2d411d7c25d463a89ce25c7"
server_key_iv: "f5503b0ea6a33a3008ddc728fe849bbf"
server_key_version: "encryptionkey@0"
I20260928 15:49:12.343621 31842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:12.369592 31842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:12.374261 31842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:12.374583 31842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:12.375623 31842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:12.377225 31842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:12.377331 31842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:12.377501 31842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:12.377583 31842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:12.412284 31842 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:34219
I20260928 15:49:12.412415 31971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:34219 every 8 connection(s)
I20260928 15:49:12.413736 31842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:49:12.417917 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 31842
I20260928 15:49:12.418160 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:49:12.418682 13333 external_mini_cluster.cc:1428] Setting key 0608606298fe3bfde8776c10a3e40fed
Sep 28 15:49:12 dist-test-slave-j968 krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610552, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:12.436203 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:12.416572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:37259 (local address 127.13.5.126:41649)
0928 15:49:12.417054 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:12.417059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:12.417936 (+ 877us) server_negotiation.cc:407] Connection header received
0928 15:49:12.420060 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:12.420065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:12.420131 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:12.420226 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:12.422043 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.422779 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.424449 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.424884 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.425022 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:12.428705 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:12.428711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:12.428714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:12.428780 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:12.432397 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:12.433289 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:12.433298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:12.433304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:12.433472 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:12.434028 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:12.434036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:12.434042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:12.434440 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:12.434609 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:12.435372 (+ 763us) server_negotiation.cc:299] Negotiation successful
0928 15:49:12.435663 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260928 15:49:12.436968 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:12.417000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:41649 (local address 127.13.5.66:37259)
0928 15:49:12.417761 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:12.417803 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:12.419741 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:12.420514 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:12.420526 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:12.421222 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:12.421845 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:12.421864 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.422950 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.422955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:12.424299 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:12.424312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.425140 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.425148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:12.425384 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:12.426114 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:12.426138 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:12.428439 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:12.432602 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:12.432609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:12.432625 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:12.433133 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:12.433643 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:12.433648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:12.433650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:12.433878 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:12.434602 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:12.434609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:12.434958 (+ 349us) client_negotiation.cc:769] Sending connection context
0928 15:49:12.435203 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 15:49:12.435685 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":278,"threads_started":1}
I20260928 15:49:12.438558 31972 heartbeater.cc:347] Connected to a master server at 127.13.5.126:41649
I20260928 15:49:12.438906 31972 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:12.439816 31972 heartbeater.cc:511] Master 127.13.5.126:41649 requested a full tablet report, sending...
I20260928 15:49:12.441587 31644 ts_manager.cc:194] Registered new tserver with Master: 8273f12559444f75a1b3555dfd5533c9 (127.13.5.66:34219)
I20260928 15:49:12.442601 31644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:37259
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:49:12.481120 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41649
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:12.818070 31978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:12.818382 31978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:12.818477 31978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:12.826452 31978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:12.826589 31978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:12.826707 31978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:49:12.837019 31978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:41649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:12.838341 31978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:12.839659 31978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:12.854137 31983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:12.854158 31986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:12.854192 31984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:12.855115 31978 server_base.cc:1034] running on GCE node
I20260928 15:49:12.855782 31978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:12.856841 31978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:12.858071 31978 hybrid_clock.cc:648] HybridClock initialized: now 1790610552858004 us; error 58 us; skew 500 ppm
I20260928 15:49:12.858440 31978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:12 dist-test-slave-j968 krb5kdc[31597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610552, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:12.862128 31978 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:49:12.865311 31978 webserver.cc:460] Webserver started at http://127.13.5.67:41745/ using document root <none> and password file <none>
I20260928 15:49:12.866222 31978 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:12.866374 31978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:12.866736 31978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:12.870252 31978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "fb0f0ddeca304d0a99f66932d0a04e39"
format_stamp: "Formatted at 2026-09-28 15:49:12 on dist-test-slave-j968"
server_key: "045565ad438e624b067edaf0a19d8c5c"
server_key_iv: "6a6d6c6c60ec0cec95695075f7dfd408"
server_key_version: "encryptionkey@0"
I20260928 15:49:12.870954 31978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "fb0f0ddeca304d0a99f66932d0a04e39"
format_stamp: "Formatted at 2026-09-28 15:49:12 on dist-test-slave-j968"
server_key: "045565ad438e624b067edaf0a19d8c5c"
server_key_iv: "6a6d6c6c60ec0cec95695075f7dfd408"
server_key_version: "encryptionkey@0"
I20260928 15:49:12.876187 31978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:49:12.879827 31993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:12.881251 31978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:12.881436 31978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "fb0f0ddeca304d0a99f66932d0a04e39"
format_stamp: "Formatted at 2026-09-28 15:49:12 on dist-test-slave-j968"
server_key: "045565ad438e624b067edaf0a19d8c5c"
server_key_iv: "6a6d6c6c60ec0cec95695075f7dfd408"
server_key_version: "encryptionkey@0"
I20260928 15:49:12.881640 31978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:12.896203 31978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:12.900255 31978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:12.900290 31836 heartbeater.cc:503] Master 127.13.5.126:41649 was elected leader, sending a full tablet report...
I20260928 15:49:12.900549 31978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:12.901652 31978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:12.903342 31978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:12.903538 31978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:12.903699 31978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:12.903777 31978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:12.941502 31978 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:33755
I20260928 15:49:12.941598 32107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:33755 every 8 connection(s)
I20260928 15:49:12.942847 31978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:49:12.946667 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 31978
I20260928 15:49:12.946933 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:49:12.947664 13333 external_mini_cluster.cc:1428] Setting key 2e7f4f8769a448612c54f0da8bb7a676
Sep 28 15:49:12 dist-test-slave-j968 krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610552, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:12.965317 32110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:12.945558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:47705 (local address 127.13.5.126:41649)
0928 15:49:12.946107 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:12.946112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:12.947685 (+ 1573us) server_negotiation.cc:407] Connection header received
0928 15:49:12.950206 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:12.950211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:12.950279 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:12.950364 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:12.952200 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.953031 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.954674 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.955085 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.955179 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:12.958754 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:12.958758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:12.958774 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:12.958809 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:12.961914 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:12.962801 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:12.962805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:12.962808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:12.962931 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:12.963449 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:12.963454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:12.963457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:12.963730 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:12.963925 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:12.964676 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 15:49:12.964896 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":307,"threads_started":1}
I20260928 15:49:12.966226 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:12.946059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:41649 (local address 127.13.5.67:47705)
0928 15:49:12.947464 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:12.947522 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:12.949850 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:12.950680 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:12.950693 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:12.951349 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:12.952042 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:12.952063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.953200 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.953204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:12.954533 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:12.954546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.955284 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.955289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:12.955575 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:12.956309 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:12.956331 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:12.958547 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:12.962055 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:12.962063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:12.962079 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:12.962673 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:12.963054 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:12.963059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:12.963063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:12.963337 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:12.963859 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:12.963887 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:12.964294 (+ 407us) client_negotiation.cc:769] Sending connection context
0928 15:49:12.964568 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 15:49:12.965097 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":847,"threads_started":1}
I20260928 15:49:12.967545 32108 heartbeater.cc:347] Connected to a master server at 127.13.5.126:41649
I20260928 15:49:12.967856 32108 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:12.968668 32108 heartbeater.cc:511] Master 127.13.5.126:41649 requested a full tablet report, sending...
I20260928 15:49:12.970013 31644 ts_manager.cc:194] Registered new tserver with Master: fb0f0ddeca304d0a99f66932d0a04e39 (127.13.5.67:33755)
I20260928 15:49:12.970894 31644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:47705
I20260928 15:49:12.975577 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:49:12.992717 32110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:12.979926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.13.5.126:41649)
0928 15:49:12.980108 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:12.980112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:12.980631 (+ 519us) server_negotiation.cc:407] Connection header received
0928 15:49:12.980782 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:12.980785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:12.980845 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:12.980924 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:12.982323 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.983020 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.984413 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:12.984811 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:12.984900 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:12.987554 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:12.987558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:12.987561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:12.987590 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:12.990213 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:12.990922 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:12.990926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:12.990928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:12.991076 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:12.991450 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:12.991453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:12.991456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:12.991761 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:12.992044 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:12.992365 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 15:49:12.992496 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:49:12.996579 31644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:49:13.003492 31644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:49:13.026211 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.015832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:34219 (local address 127.0.0.1:47448)
0928 15:49:13.016695 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:13.016735 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:13.016949 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:13.017787 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:13.017795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:13.017828 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:13.018269 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.018284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.020539 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.020545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.023143 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.023160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.025093 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.025101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.025308 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.025366 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 15:49:13.025480 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:49:13.025693 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":1,"thread_start_us":385,"threads_started":1}
I20260928 15:49:13.026460 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.017032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43323 (local address 127.0.0.1:52524)
0928 15:49:13.017593 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:13.017612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:13.017727 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:13.018976 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:13.018980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:13.019006 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:13.019420 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.019432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.021572 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.021577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.023695 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.023709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.025635 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.025641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.025823 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.025865 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:49:13.025932 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 15:49:13.026058 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":269,"threads_started":1}
I20260928 15:49:13.027529 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.017441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.13.5.65:43323)
0928 15:49:13.018287 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.018296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.018327 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:49:13.018427 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.018437 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.018755 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.019613 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:13.019887 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.021398 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.023922 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.025392 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.026676 (+ 1284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.026952 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.027018 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.027186 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":496,"threads_started":1}
I20260928 15:49:13.027926 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.015997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.13.5.66:34219)
0928 15:49:13.016609 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.016620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.016834 (+ 214us) server_negotiation.cc:407] Connection header received
0928 15:49:13.017259 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.017271 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.017552 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.018499 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:13.018754 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.020385 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.023359 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.024838 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.027144 (+ 2306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.027337 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.027407 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.027580 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":320,"threads_started":1}
I20260928 15:49:13.032589 31907 tablet_service.cc:1467] Processing CreateTablet for tablet f3def694329e4a3a9c2d98db5f82e8ff (DEFAULT_TABLE table=test-table [id=0fdfeac78e174e278deb6fa84ad80428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:13.032586 31771 tablet_service.cc:1467] Processing CreateTablet for tablet f3def694329e4a3a9c2d98db5f82e8ff (DEFAULT_TABLE table=test-table [id=0fdfeac78e174e278deb6fa84ad80428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:13.034150 31771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3def694329e4a3a9c2d98db5f82e8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:13.034152 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3def694329e4a3a9c2d98db5f82e8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:13.034567 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.018301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:33755 (local address 127.0.0.1:43592)
0928 15:49:13.026475 (+ 8174us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:13.026495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:13.026603 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:13.029133 (+ 2530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:13.029138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:13.029165 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:13.029329 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.029339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.030825 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.030830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.032453 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.032464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.033772 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.033778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.033958 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.033991 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:49:13.034090 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:49:13.034231 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8056,"thread_start_us":8622,"threads_started":1}
I20260928 15:49:13.035060 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.018001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.13.5.67:33755)
0928 15:49:13.028419 (+ 10418us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.028430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.028470 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:49:13.028609 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.028619 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.028914 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.029128 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:13.029507 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.030649 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.032609 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.033613 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.033759 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.033980 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.034147 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.034306 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10246,"thread_start_us":10084,"threads_started":1}
I20260928 15:49:13.038558 32042 tablet_service.cc:1467] Processing CreateTablet for tablet f3def694329e4a3a9c2d98db5f82e8ff (DEFAULT_TABLE table=test-table [id=0fdfeac78e174e278deb6fa84ad80428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:13.040205 32042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3def694329e4a3a9c2d98db5f82e8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:13.049634 32128 tablet_bootstrap.cc:492] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Bootstrap starting.
I20260928 15:49:13.050834 32127 tablet_bootstrap.cc:492] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Bootstrap starting.
I20260928 15:49:13.052876 32129 tablet_bootstrap.cc:492] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: Bootstrap starting.
I20260928 15:49:13.054472 32127 tablet_bootstrap.cc:654] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:13.054867 32128 tablet_bootstrap.cc:654] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:13.055581 32127 log.cc:824] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:13.056098 32128 log.cc:824] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:13.056339 32129 tablet_bootstrap.cc:654] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:13.057580 32129 log.cc:824] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:13.059131 32127 tablet_bootstrap.cc:492] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: No bootstrap required, opened a new log
I20260928 15:49:13.059183 32128 tablet_bootstrap.cc:492] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: No bootstrap required, opened a new log
I20260928 15:49:13.059538 32127 ts_tablet_manager.cc:1397] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 15:49:13.059551 32128 ts_tablet_manager.cc:1397] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 15:49:13.061829 32129 tablet_bootstrap.cc:492] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: No bootstrap required, opened a new log
I20260928 15:49:13.062111 32129 ts_tablet_manager.cc:1397] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 15:49:13.067262 32128 raft_consensus.cc:348] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.067888 32128 raft_consensus.cc:374] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:13.068091 32128 raft_consensus.cc:729] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8273f12559444f75a1b3555dfd5533c9, State: Initialized, Role: FOLLOWER
I20260928 15:49:13.068804 32128 consensus_queue.cc:260] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.069927 31972 heartbeater.cc:503] Master 127.13.5.126:41649 was elected leader, sending a full tablet report...
I20260928 15:49:13.070504 32127 raft_consensus.cc:348] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.070946 32127 raft_consensus.cc:374] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:13.071103 32127 raft_consensus.cc:729] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00124c84ea0e4fc4ba042f879d3385a8, State: Initialized, Role: FOLLOWER
I20260928 15:49:13.071961 32128 ts_tablet_manager.cc:1428] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260928 15:49:13.071882 32127 consensus_queue.cc:260] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.073771 32127 ts_tablet_manager.cc:1428] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 15:49:13.072806 32129 raft_consensus.cc:348] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.074111 32129 raft_consensus.cc:374] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:13.074260 32129 raft_consensus.cc:729] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0f0ddeca304d0a99f66932d0a04e39, State: Initialized, Role: FOLLOWER
I20260928 15:49:13.074887 32129 consensus_queue.cc:260] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.076114 32108 heartbeater.cc:503] Master 127.13.5.126:41649 was elected leader, sending a full tablet report...
I20260928 15:49:13.076577 32129 ts_tablet_manager.cc:1428] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260928 15:49:13.100235 32133 raft_consensus.cc:482] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:49:13.100586 32133 raft_consensus.cc:504] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.102308 32133 leader_election.cc:288] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00124c84ea0e4fc4ba042f879d3385a8 (127.13.5.65:43323), fb0f0ddeca304d0a99f66932d0a04e39 (127.13.5.67:33755)
I20260928 15:49:13.110237 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.103630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:51535 (local address 127.13.5.67:33755)
0928 15:49:13.103834 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.103839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.104138 (+ 299us) server_negotiation.cc:407] Connection header received
0928 15:49:13.104364 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.104369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.104446 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.104548 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:13.105027 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.105991 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.108361 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.109150 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.109253 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.109329 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.109829 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.109973 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:49:13.110306 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.102740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:49847 (local address 127.13.5.65:43323)
0928 15:49:13.103011 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.103018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.103409 (+ 391us) server_negotiation.cc:407] Connection header received
0928 15:49:13.103663 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.103669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.103756 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.103809 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:13.104565 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.105608 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.108222 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.109132 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.109260 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.109373 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.109955 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.110088 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:49:13.110447 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.103369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:33755 (local address 127.13.5.66:51535)
0928 15:49:13.104004 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:13.104048 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:13.104224 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:13.104585 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:13.104588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:13.104606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:13.104882 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.104889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.106110 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.106116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.108207 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.108219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.109374 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.109381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.109659 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.109695 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:49:13.109800 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:49:13.109950 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":22,"thread_start_us":491,"threads_started":1}
I20260928 15:49:13.110570 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.102619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43323 (local address 127.13.5.66:49847)
0928 15:49:13.103232 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:13.103279 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:13.103451 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:13.104046 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:13.104049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:13.104070 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:13.104416 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.104424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.105744 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.105747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.108021 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:13.108038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.109575 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.109580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:13.109803 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.109831 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 15:49:13.109907 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 15:49:13.110009 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":22,"spinlock_wait_cycles":29568,"thread_start_us":313,"threads_started":1}
I20260928 15:49:13.111771 31791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3def694329e4a3a9c2d98db5f82e8ff" candidate_uuid: "8273f12559444f75a1b3555dfd5533c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00124c84ea0e4fc4ba042f879d3385a8" is_pre_election: true
I20260928 15:49:13.111761 32063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3def694329e4a3a9c2d98db5f82e8ff" candidate_uuid: "8273f12559444f75a1b3555dfd5533c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" is_pre_election: true
I20260928 15:49:13.112437 32063 raft_consensus.cc:2454] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8273f12559444f75a1b3555dfd5533c9 in term 0.
I20260928 15:49:13.112437 31791 raft_consensus.cc:2454] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8273f12559444f75a1b3555dfd5533c9 in term 0.
I20260928 15:49:13.113190 31859 leader_election.cc:302] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8273f12559444f75a1b3555dfd5533c9, fb0f0ddeca304d0a99f66932d0a04e39; no voters:
I20260928 15:49:13.113631 32133 raft_consensus.cc:2781] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:49:13.113781 32133 raft_consensus.cc:482] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:49:13.113909 32133 raft_consensus.cc:3037] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:49:13.116225 31837 tablet.cc:2369] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:49:13.116266 32133 raft_consensus.cc:504] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.116930 32133 leader_election.cc:288] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [CANDIDATE]: Term 1 election: Requested vote from peers 00124c84ea0e4fc4ba042f879d3385a8 (127.13.5.65:43323), fb0f0ddeca304d0a99f66932d0a04e39 (127.13.5.67:33755)
I20260928 15:49:13.117443 31791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3def694329e4a3a9c2d98db5f82e8ff" candidate_uuid: "8273f12559444f75a1b3555dfd5533c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00124c84ea0e4fc4ba042f879d3385a8"
I20260928 15:49:13.117722 31791 raft_consensus.cc:3037] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:13.117841 32063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3def694329e4a3a9c2d98db5f82e8ff" candidate_uuid: "8273f12559444f75a1b3555dfd5533c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0f0ddeca304d0a99f66932d0a04e39"
I20260928 15:49:13.118088 32063 raft_consensus.cc:3037] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:13.120095 31791 raft_consensus.cc:2454] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8273f12559444f75a1b3555dfd5533c9 in term 1.
I20260928 15:49:13.120522 31859 leader_election.cc:302] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00124c84ea0e4fc4ba042f879d3385a8, 8273f12559444f75a1b3555dfd5533c9; no voters:
I20260928 15:49:13.120558 32063 raft_consensus.cc:2454] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8273f12559444f75a1b3555dfd5533c9 in term 1.
I20260928 15:49:13.120850 32133 raft_consensus.cc:2781] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:13.121330 32133 raft_consensus.cc:686] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 1 LEADER]: Becoming Leader. State: Replica: 8273f12559444f75a1b3555dfd5533c9, State: Running, Role: LEADER
I20260928 15:49:13.121833 32133 consensus_queue.cc:237] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } }
I20260928 15:49:13.126529 31644 catalog_manager.cc:5495] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8273f12559444f75a1b3555dfd5533c9 (127.13.5.66). New cstate: current_term: 1 leader_uuid: "8273f12559444f75a1b3555dfd5533c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8273f12559444f75a1b3555dfd5533c9" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 34219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 } health_report { overall_health: UNKNOWN } } }
I20260928 15:49:13.165284 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.160146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.13.5.66:34219)
0928 15:49:13.160338 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.160342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.160361 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:49:13.160444 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.160447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.160530 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.160658 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:49:13.161030 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.161752 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.163061 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.163443 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.163538 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.164120 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:49:13.164708 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:49:13.164865 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.165023 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.165066 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 15:49:13.166138 31973 tablet.cc:2369] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:49:13.173619 32063 raft_consensus.cc:1261] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 1 FOLLOWER]: Refusing update from remote peer 8273f12559444f75a1b3555dfd5533c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:13.173753 31791 raft_consensus.cc:1261] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 1 FOLLOWER]: Refusing update from remote peer 8273f12559444f75a1b3555dfd5533c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:13.174656 32138 consensus_queue.cc:1035] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0f0ddeca304d0a99f66932d0a04e39" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 33755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:13.175001 32133 consensus_queue.cc:1035] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00124c84ea0e4fc4ba042f879d3385a8" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:13.187326 32141 mvcc.cc:205] Tried to move back new op lower bound from 7334340825789063168 to 7334340825591316480. Current Snapshot: MvccSnapshot[applied={T|T < 7334340825789063168}]
I20260928 15:49:13.190828 32143 mvcc.cc:205] Tried to move back new op lower bound from 7334340825789063168 to 7334340825591316480. Current Snapshot: MvccSnapshot[applied={T|T < 7334340825789063168}]
I20260928 15:49:13.201570 31642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:13.201894 31642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:13.205785 31642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e7b2245cbd844f5a5cab0240fa34e12: Sending DeleteTablet for 3 replicas of tablet f3def694329e4a3a9c2d98db5f82e8ff
I20260928 15:49:13.207062 31907 tablet_service.cc:1514] Processing DeleteTablet for tablet f3def694329e4a3a9c2d98db5f82e8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:13 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:47448
I20260928 15:49:13.207175 31771 tablet_service.cc:1514] Processing DeleteTablet for tablet f3def694329e4a3a9c2d98db5f82e8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:13 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52524
I20260928 15:49:13.207355 32042 tablet_service.cc:1514] Processing DeleteTablet for tablet f3def694329e4a3a9c2d98db5f82e8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:13 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:43592
I20260928 15:49:13.208951 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 31706
I20260928 15:49:13.219655 32152 tablet_replica.cc:330] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: stopping tablet replica
I20260928 15:49:13.220324 32151 tablet_replica.cc:330] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: stopping tablet replica
I20260928 15:49:13.221017 32151 raft_consensus.cc:2229] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:13.221624 32152 raft_consensus.cc:2229] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:13.222216 32152 raft_consensus.cc:2258] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:13.222715 32151 raft_consensus.cc:2258] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:13.225317 32152 ts_tablet_manager.cc:1905] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:13.232157 32151 ts_tablet_manager.cc:1905] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:13.233888 32152 ts_tablet_manager.cc:1918] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:13.234233 32152 log.cc:1196] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/f3def694329e4a3a9c2d98db5f82e8ff
I20260928 15:49:13.235271 32152 ts_tablet_manager.cc:1939] T f3def694329e4a3a9c2d98db5f82e8ff P fb0f0ddeca304d0a99f66932d0a04e39: Deleting consensus metadata
I20260928 15:49:13.236786 32150 tablet_replica.cc:330] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: stopping tablet replica
I20260928 15:49:13.237447 32150 raft_consensus.cc:2229] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:49:13.237622 31629 catalog_manager.cc:4841] TS fb0f0ddeca304d0a99f66932d0a04e39 (127.13.5.67:33755): tablet f3def694329e4a3a9c2d98db5f82e8ff (table test-table [id=0fdfeac78e174e278deb6fa84ad80428]) successfully deleted
I20260928 15:49:13.238359 32150 raft_consensus.cc:2258] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:13.240618 32150 ts_tablet_manager.cc:1905] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:49:13 dist-test-slave-j968 krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:49:13.248970 32150 ts_tablet_manager.cc:1918] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:13.249063 32151 ts_tablet_manager.cc:1918] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:13.249310 32151 log.cc:1196] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/f3def694329e4a3a9c2d98db5f82e8ff
I20260928 15:49:13.249375 32150 log.cc:1196] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/f3def694329e4a3a9c2d98db5f82e8ff
I20260928 15:49:13.249957 32151 ts_tablet_manager.cc:1939] T f3def694329e4a3a9c2d98db5f82e8ff P 00124c84ea0e4fc4ba042f879d3385a8: Deleting consensus metadata
I20260928 15:49:13.250048 32150 ts_tablet_manager.cc:1939] T f3def694329e4a3a9c2d98db5f82e8ff P 8273f12559444f75a1b3555dfd5533c9: Deleting consensus metadata
I20260928 15:49:13.251945 31629 catalog_manager.cc:4841] TS 00124c84ea0e4fc4ba042f879d3385a8 (127.13.5.65:43323): tablet f3def694329e4a3a9c2d98db5f82e8ff (table test-table [id=0fdfeac78e174e278deb6fa84ad80428]) successfully deleted
I20260928 15:49:13.252292 31629 catalog_manager.cc:4841] TS 8273f12559444f75a1b3555dfd5533c9 (127.13.5.66:34219): tablet f3def694329e4a3a9c2d98db5f82e8ff (table test-table [id=0fdfeac78e174e278deb6fa84ad80428]) successfully deleted
I20260928 15:49:13.253000 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.211121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.13.5.65:43323)
0928 15:49:13.211373 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.211382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.222459 (+ 11077us) server_negotiation.cc:407] Connection header received
0928 15:49:13.223037 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.223043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.223155 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.223242 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:13.225208 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.226396 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.228930 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.229617 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.229721 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.244293 (+ 14572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:13.244315 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:13.244338 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:13.244399 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:13.248701 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:13.249989 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:13.250001 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:13.250005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:13.250199 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:13.250731 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:13.250738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:13.250742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:13.251261 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:13.251570 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.252391 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.252634 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:49:13.253691 31831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:13.362447 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 31706
I20260928 15:49:13.387918 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 31842
Sep 28 15:49:13 dist-test-slave-j968 krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:49:13.412454 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.388889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.13.5.66:34219)
0928 15:49:13.389123 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.389129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.389149 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:49:13.389223 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.389227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.389310 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.389423 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:13.391718 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.393462 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.395488 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.396432 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.396588 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.402688 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:13.402725 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:13.402756 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:13.402817 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:13.408270 (+ 5453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:13.409166 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:13.409178 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:13.409183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:13.409370 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:13.409883 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:13.409888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:13.409891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:13.410451 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:13.410818 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.411772 (+ 954us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.412046 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:49:13.413542 31967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:13.537186 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 31842
I20260928 15:49:13.557806 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 31978
Sep 28 15:49:13 dist-test-slave-j968 krb5kdc[31597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:49:13.573213 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:13.558605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.13.5.67:33755)
0928 15:49:13.558823 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:13.558827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:13.558869 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:49:13.559044 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:13.559047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:13.559117 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:13.559200 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:13.560671 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.561529 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.562790 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:13.563200 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:13.563318 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:13.568008 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:13.568023 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:13.568041 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:13.568083 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:13.570662 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:13.571324 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:13.571330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:13.571333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:13.571470 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:13.571810 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:13.571813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:13.571815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:13.572238 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:13.572458 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:13.572771 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 15:49:13.572946 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:49:13.573907 32103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:13.664397 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 31978
I20260928 15:49:13.683070 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 31610
I20260928 15:49:13.684319 31675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:13.797951 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 31610
2026-09-28T15:49:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4422 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T15:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:13Z Disabled control of system clock
I20260928 15:49:13.878614 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:33237
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:33237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:49:14.238647 32161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:14.238948 32161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:14.239100 32161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:14.246795 32161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:14.246902 32161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:14.246968 32161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:14.247056 32161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:14.247134 32161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:14.256600 32161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:33237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:33237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:14.257936 32161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:14.260056 32161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:14.272068 32166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:14.272164 32167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:14.272262 32169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:14.272863 32161 server_base.cc:1034] running on GCE node
I20260928 15:49:14.273802 32161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:14.275251 32161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:14.276500 32161 hybrid_clock.cc:648] HybridClock initialized: now 1790610554276444 us; error 57 us; skew 500 ppm
I20260928 15:49:14.276857 32161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:14.279316 32161 webserver.cc:460] Webserver started at http://127.13.5.126:38795/ using document root <none> and password file <none>
I20260928 15:49:14.280129 32161 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:14.280262 32161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:14.280596 32161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:14.283538 32161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "f7ab296d652d471a9b271a31238c70cf"
format_stamp: "Formatted at 2026-09-28 15:49:14 on dist-test-slave-j968"
server_key: "2ea398b25605148d988fc80d742ee7d6"
server_key_iv: "1e9f93d5d910e1cbea74b724ae22c7ec"
server_key_version: "encryptionkey@0"
I20260928 15:49:14.284421 32161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "f7ab296d652d471a9b271a31238c70cf"
format_stamp: "Formatted at 2026-09-28 15:49:14 on dist-test-slave-j968"
server_key: "2ea398b25605148d988fc80d742ee7d6"
server_key_iv: "1e9f93d5d910e1cbea74b724ae22c7ec"
server_key_version: "encryptionkey@0"
I20260928 15:49:14.289791 32161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:49:14.293419 32175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:14.294835 32161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:14.295060 32161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "f7ab296d652d471a9b271a31238c70cf"
format_stamp: "Formatted at 2026-09-28 15:49:14 on dist-test-slave-j968"
server_key: "2ea398b25605148d988fc80d742ee7d6"
server_key_iv: "1e9f93d5d910e1cbea74b724ae22c7ec"
server_key_version: "encryptionkey@0"
I20260928 15:49:14.295280 32161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:14.328859 32161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:14.329815 32161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:14.330128 32161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:14.349933 32161 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:33237
I20260928 15:49:14.349954 32227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:33237 every 8 connection(s)
I20260928 15:49:14.351353 32161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:14.354235 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 32161
I20260928 15:49:14.354445 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:14.355038 13333 external_mini_cluster.cc:1428] Setting key 0489b2987c2f3ea7b2a5e2275e04cdfc
I20260928 15:49:14.355973 32228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:14.370114 32232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:14.357128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.13.5.126:33237)
0928 15:49:14.358146 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:14.358158 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:14.358200 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:49:14.360649 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:14.360671 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:14.361207 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:14.361577 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:14.362210 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:14.363582 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:14.365506 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:14.366186 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:14.366328 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:14.367621 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:14.367628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:14.367646 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:14.368357 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:14.368456 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:14.368479 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:14.368569 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:14.368796 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 15:49:14.369176 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":408,"threads_started":1}
I20260928 15:49:14.370376 32228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf: Bootstrap starting.
I20260928 15:49:14.373570 32228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:14.374565 32228 log.cc:824] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:14.377621 32228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf: No bootstrap required, opened a new log
I20260928 15:49:14.384961 32228 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ab296d652d471a9b271a31238c70cf" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33237 } }
I20260928 15:49:14.385365 32228 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:14.385478 32228 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ab296d652d471a9b271a31238c70cf, State: Initialized, Role: FOLLOWER
I20260928 15:49:14.386052 32228 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ab296d652d471a9b271a31238c70cf" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33237 } }
I20260928 15:49:14.386296 32228 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:14.386438 32228 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:14.386618 32228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:14.389298 32228 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ab296d652d471a9b271a31238c70cf" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33237 } }
I20260928 15:49:14.389822 32228 leader_election.cc:302] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7ab296d652d471a9b271a31238c70cf; no voters:
I20260928 15:49:14.390412 32228 leader_election.cc:288] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:14.390609 32234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:14.391357 32234 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [term 1 LEADER]: Becoming Leader. State: Replica: f7ab296d652d471a9b271a31238c70cf, State: Running, Role: LEADER
I20260928 15:49:14.391947 32234 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ab296d652d471a9b271a31238c70cf" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33237 } }
I20260928 15:49:14.392663 32228 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:14.395006 32236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7ab296d652d471a9b271a31238c70cf. Latest consensus state: current_term: 1 leader_uuid: "f7ab296d652d471a9b271a31238c70cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ab296d652d471a9b271a31238c70cf" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33237 } } }
I20260928 15:49:14.395555 32236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:14.395117 32235 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7ab296d652d471a9b271a31238c70cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ab296d652d471a9b271a31238c70cf" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33237 } } }
I20260928 15:49:14.395906 32235 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:14.397365 32243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:14.403043 32243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:14.412302 32243 catalog_manager.cc:1347] Generated new cluster ID: 9e4664b18fb042c2b936058fca718059
I20260928 15:49:14.412441 32243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:14.432926 32243 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:14.434496 32243 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:14.456660 32243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf: Generated new TSK 0
I20260928 15:49:14.457597 32243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:14.469156 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33237
--builtin_ntp_servers=127.13.5.84:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:49:14.813118 32253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:14.813441 32253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:14.813616 32253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:14.820916 32253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:14.821085 32253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:14.821265 32253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:49:14.830899 32253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:14.832356 32253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:14.834466 32253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:14.848440 32258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:14.848650 32261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:14.848657 32259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:14.849713 32253 server_base.cc:1034] running on GCE node
I20260928 15:49:14.850409 32253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:14.851436 32253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:14.852682 32253 hybrid_clock.cc:648] HybridClock initialized: now 1790610554852627 us; error 48 us; skew 500 ppm
I20260928 15:49:14.853127 32253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:14.855443 32253 webserver.cc:460] Webserver started at http://127.13.5.65:33717/ using document root <none> and password file <none>
I20260928 15:49:14.856245 32253 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:14.856371 32253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:14.856703 32253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:14.859676 32253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "826283a3feee430497a1998e68697a1c"
format_stamp: "Formatted at 2026-09-28 15:49:14 on dist-test-slave-j968"
server_key: "26d0a7406aaee1f2d1fd952a728019a7"
server_key_iv: "032612bbc11eb8cc92229244a61e870a"
server_key_version: "encryptionkey@0"
I20260928 15:49:14.860540 32253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "826283a3feee430497a1998e68697a1c"
format_stamp: "Formatted at 2026-09-28 15:49:14 on dist-test-slave-j968"
server_key: "26d0a7406aaee1f2d1fd952a728019a7"
server_key_iv: "032612bbc11eb8cc92229244a61e870a"
server_key_version: "encryptionkey@0"
I20260928 15:49:14.865870 32253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:49:14.869524 32267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:14.870839 32253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 15:49:14.871032 32253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "826283a3feee430497a1998e68697a1c"
format_stamp: "Formatted at 2026-09-28 15:49:14 on dist-test-slave-j968"
server_key: "26d0a7406aaee1f2d1fd952a728019a7"
server_key_iv: "032612bbc11eb8cc92229244a61e870a"
server_key_version: "encryptionkey@0"
I20260928 15:49:14.871253 32253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:14.914498 32253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:14.915491 32253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:14.915813 32253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:14.916939 32253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:14.918617 32253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:14.918720 32253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:14.918817 32253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:14.918885 32253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:14.955617 32253 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:38257
I20260928 15:49:14.955650 32382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:38257 every 8 connection(s)
I20260928 15:49:14.957222 32253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:49:14.965368 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 32253
I20260928 15:49:14.965603 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:49:14.966073 13333 external_mini_cluster.cc:1428] Setting key 0cfa8d6a4084cbd8fbd7bf0058aa338d
I20260928 15:49:14.969285 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33237
--builtin_ntp_servers=127.13.5.84:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:49:14.970635 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:14.959910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:40719 (local address 127.13.5.126:33237)
0928 15:49:14.960359 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:14.960363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:14.961677 (+ 1314us) server_negotiation.cc:407] Connection header received
0928 15:49:14.963518 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:14.963524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:14.963623 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:14.963750 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:14.964866 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:14.965698 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:14.967529 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:14.968093 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:14.968226 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:14.969816 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:14.969821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:14.969824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:14.969842 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:14.969881 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:14.969892 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:14.969968 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:14.970232 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 15:49:14.970310 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":217,"threads_started":1}
I20260928 15:49:14.972162 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:14.960540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33237 (local address 127.13.5.65:40719)
0928 15:49:14.961508 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:14.961547 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:14.963236 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:14.964022 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:14.964031 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:14.964121 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:14.964689 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:14.964705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:14.965871 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:14.965875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:14.967379 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:14.967392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:14.968352 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:14.968360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:14.968681 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:14.969426 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:14.969431 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:14.969512 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:14.969623 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:14.969993 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:14.970007 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:14.970019 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:49:14.970226 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 15:49:14.970687 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":270,"threads_started":1}
I20260928 15:49:14.973764 32383 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33237
I20260928 15:49:14.974087 32383 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:14.975085 32383 heartbeater.cc:511] Master 127.13.5.126:33237 requested a full tablet report, sending...
I20260928 15:49:14.977545 32192 ts_manager.cc:194] Registered new tserver with Master: 826283a3feee430497a1998e68697a1c (127.13.5.65:38257)
I20260928 15:49:14.979486 32192 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.65:40719
W20260928 15:49:15.307080 32387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:15.307372 32387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:15.307471 32387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:15.314774 32387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:15.314879 32387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:15.314985 32387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:49:15.324231 32387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:15.325511 32387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:15.327244 32387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:15.341315 32393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:15.341313 32394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:15.341230 32396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:15.342516 32387 server_base.cc:1034] running on GCE node
I20260928 15:49:15.343139 32387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:15.344103 32387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:15.345315 32387 hybrid_clock.cc:648] HybridClock initialized: now 1790610555345274 us; error 27 us; skew 500 ppm
I20260928 15:49:15.345708 32387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:15.348138 32387 webserver.cc:460] Webserver started at http://127.13.5.66:43825/ using document root <none> and password file <none>
I20260928 15:49:15.348887 32387 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:15.348999 32387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:15.349349 32387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:15.352268 32387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "34bbaba0ca98411db0746820da79f684"
format_stamp: "Formatted at 2026-09-28 15:49:15 on dist-test-slave-j968"
server_key: "46e736b61ff3b6e47d0e14d052e5330a"
server_key_iv: "3e96e8cbcc58ad6976eae2b2caefdcaf"
server_key_version: "encryptionkey@0"
I20260928 15:49:15.353067 32387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "34bbaba0ca98411db0746820da79f684"
format_stamp: "Formatted at 2026-09-28 15:49:15 on dist-test-slave-j968"
server_key: "46e736b61ff3b6e47d0e14d052e5330a"
server_key_iv: "3e96e8cbcc58ad6976eae2b2caefdcaf"
server_key_version: "encryptionkey@0"
I20260928 15:49:15.358335 32387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260928 15:49:15.362169 32402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:15.363662 32387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 15:49:15.363834 32387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "34bbaba0ca98411db0746820da79f684"
format_stamp: "Formatted at 2026-09-28 15:49:15 on dist-test-slave-j968"
server_key: "46e736b61ff3b6e47d0e14d052e5330a"
server_key_iv: "3e96e8cbcc58ad6976eae2b2caefdcaf"
server_key_version: "encryptionkey@0"
I20260928 15:49:15.364133 32387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:15.391947 32387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:15.392935 32387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:15.393217 32387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:15.394248 32387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:15.395953 32387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:15.396066 32387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:15.396212 32387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:15.396299 32387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:15.431143 32387 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:35587
I20260928 15:49:15.431233 32516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:35587 every 8 connection(s)
I20260928 15:49:15.432544 32387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:49:15.437780 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 32387
I20260928 15:49:15.437971 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:49:15.438395 13333 external_mini_cluster.cc:1428] Setting key 6ccd1c9c35d99cce57243efa78cf1920
I20260928 15:49:15.441495 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33237
--builtin_ntp_servers=127.13.5.84:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:49:15.445695 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.435291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:60095 (local address 127.13.5.126:33237)
0928 15:49:15.435533 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:15.435538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:15.436903 (+ 1365us) server_negotiation.cc:407] Connection header received
0928 15:49:15.438870 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:15.438875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:15.438974 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:15.439086 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:15.440467 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.441185 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.442868 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.443286 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.443421 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.444801 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:15.444806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:15.444809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:15.444826 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:15.444865 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:15.444910 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:15.445039 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:15.445272 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 15:49:15.445371 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:49:15.446696 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.435781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33237 (local address 127.13.5.66:60095)
0928 15:49:15.436732 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:15.436775 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:15.438604 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:15.439320 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:15.439329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:15.439469 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:15.440291 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.440310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.441342 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.441352 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.442715 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.442726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.443510 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.443518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.443739 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.444490 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:15.444493 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:15.444536 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:15.444610 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:15.445047 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:15.445059 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:15.445067 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:49:15.445251 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 15:49:15.445642 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":274,"threads_started":1}
I20260928 15:49:15.448192 32517 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33237
I20260928 15:49:15.448518 32517 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:15.449451 32517 heartbeater.cc:511] Master 127.13.5.126:33237 requested a full tablet report, sending...
I20260928 15:49:15.451093 32192 ts_manager.cc:194] Registered new tserver with Master: 34bbaba0ca98411db0746820da79f684 (127.13.5.66:35587)
I20260928 15:49:15.451972 32192 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.66:60095
W20260928 15:49:15.778349 32520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:15.778709 32520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:15.778877 32520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:15.786237 32520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:15.786384 32520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:15.786568 32520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:49:15.795976 32520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:33237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:15.797370 32520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:15.799256 32520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:15.813160 32528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:15.813325 32525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:15.813355 32526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:15.814366 32520 server_base.cc:1034] running on GCE node
I20260928 15:49:15.815019 32520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:15.816004 32520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:15.817212 32520 hybrid_clock.cc:648] HybridClock initialized: now 1790610555817138 us; error 65 us; skew 500 ppm
I20260928 15:49:15.817615 32520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:15.820071 32520 webserver.cc:460] Webserver started at http://127.13.5.67:41423/ using document root <none> and password file <none>
I20260928 15:49:15.820822 32520 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:15.820938 32520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:15.821285 32520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:15.824317 32520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "10d6300ea6084093a67828b14321aad0"
format_stamp: "Formatted at 2026-09-28 15:49:15 on dist-test-slave-j968"
server_key: "244c99fc41486fcace423673937adeda"
server_key_iv: "58bb0147e65a21e1b1eb4b47875bb2a9"
server_key_version: "encryptionkey@0"
I20260928 15:49:15.825145 32520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "10d6300ea6084093a67828b14321aad0"
format_stamp: "Formatted at 2026-09-28 15:49:15 on dist-test-slave-j968"
server_key: "244c99fc41486fcace423673937adeda"
server_key_iv: "58bb0147e65a21e1b1eb4b47875bb2a9"
server_key_version: "encryptionkey@0"
I20260928 15:49:15.830783 32520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:49:15.834347 32534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:15.835951 32520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:15.836120 32520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "10d6300ea6084093a67828b14321aad0"
format_stamp: "Formatted at 2026-09-28 15:49:15 on dist-test-slave-j968"
server_key: "244c99fc41486fcace423673937adeda"
server_key_iv: "58bb0147e65a21e1b1eb4b47875bb2a9"
server_key_version: "encryptionkey@0"
I20260928 15:49:15.836350 32520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:15.854171 32520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:15.855057 32520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:15.855351 32520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:15.856458 32520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:15.858011 32520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:15.858126 32520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:15.858253 32520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:15.858343 32520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:15.891814 32520 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:39967
I20260928 15:49:15.891939 32648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:39967 every 8 connection(s)
I20260928 15:49:15.893680 32520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:49:15.898175 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 32520
I20260928 15:49:15.898371 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:49:15.898897 13333 external_mini_cluster.cc:1428] Setting key 0e66b3d66b6245e0e4681c59b950f4f0
I20260928 15:49:15.907630 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.896551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:57503 (local address 127.13.5.126:33237)
0928 15:49:15.896773 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:15.896780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:15.898341 (+ 1561us) server_negotiation.cc:407] Connection header received
0928 15:49:15.900883 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:15.900889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:15.900989 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:15.901101 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:15.902349 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.903136 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.904754 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.905159 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.905293 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.906734 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:15.906740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:15.906743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:15.906762 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:15.906793 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:15.906805 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:15.906933 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:15.907185 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 15:49:15.907288 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:49:15.908711 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.897193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:33237 (local address 127.13.5.67:57503)
0928 15:49:15.898164 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:15.898206 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:15.900578 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:15.901356 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:15.901364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:15.901473 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:15.902184 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.902202 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.903288 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.903293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.904605 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.904617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.905382 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.905390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.905618 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.906305 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:15.906308 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:15.906353 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:15.906514 (+ 161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:15.906933 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:15.906946 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:15.906967 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 15:49:15.907175 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 15:49:15.907598 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":292,"threads_started":1}
I20260928 15:49:15.910033 32649 heartbeater.cc:347] Connected to a master server at 127.13.5.126:33237
I20260928 15:49:15.910337 32649 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:15.911131 32649 heartbeater.cc:511] Master 127.13.5.126:33237 requested a full tablet report, sending...
I20260928 15:49:15.912863 32192 ts_manager.cc:194] Registered new tserver with Master: 10d6300ea6084093a67828b14321aad0 (127.13.5.67:39967)
I20260928 15:49:15.913918 32192 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.67:57503
I20260928 15:49:15.914710 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:49:15.925024 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.918884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.13.5.126:33237)
0928 15:49:15.919162 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:15.919169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:15.919632 (+ 463us) server_negotiation.cc:407] Connection header received
0928 15:49:15.919714 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:15.919719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:15.919804 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:15.920005 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:15.920481 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.921187 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.922582 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.923000 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.923105 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.924242 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:15.924248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:15.924251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:15.924269 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:15.924302 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:15.924314 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:15.924429 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:15.924619 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 15:49:15.924697 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 15:49:15.929136 32191 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:49:15.936414 32191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:49:15.978637 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.949177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55266)
0928 15:49:15.969855 (+ 20678us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:15.969874 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:15.970026 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:15.971209 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:15.971217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:15.971252 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:15.971775 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.971790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.973831 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.973838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.976048 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.976061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.977675 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.977682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.977912 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.977984 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 15:49:15.978109 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 15:49:15.978300 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20518,"thread_start_us":21924,"threads_started":1}
I20260928 15:49:15.979627 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.949789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.13.5.65:38257)
0928 15:49:15.969385 (+ 19596us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:15.969393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:15.970004 (+ 611us) server_negotiation.cc:407] Connection header received
0928 15:49:15.970231 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:15.970243 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:15.970611 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:15.970853 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:15.972027 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.973639 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.976238 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.977471 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.977609 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.977911 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:15.979117 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0928 15:49:15.979253 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19408,"thread_start_us":19273,"threads_started":1}
I20260928 15:49:15.979846 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.950727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:35587 (local address 127.0.0.1:58594)
0928 15:49:15.971445 (+ 20718us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:15.971468 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:15.971603 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:15.972282 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:15.972286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:15.972313 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:15.972756 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.972770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.974534 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.974542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.976804 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.976819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.979001 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.979008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.979165 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.979199 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:49:15.979297 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:49:15.979452 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20548,"mutex_wait_us":34,"thread_start_us":19959,"threads_started":1}
I20260928 15:49:15.980119 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.951070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.13.5.66:35587)
0928 15:49:15.963376 (+ 12306us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:15.963385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:15.971744 (+ 8359us) server_negotiation.cc:407] Connection header received
0928 15:49:15.971847 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:15.971852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:15.972097 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:15.972294 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:15.972938 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.974392 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.977016 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.978512 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.978639 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.978860 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:15.979373 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 15:49:15.979550 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12093,"thread_start_us":12309,"threads_started":1}
I20260928 15:49:15.980379 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.949050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39967 (local address 127.0.0.1:38818)
0928 15:49:15.950253 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:15.950311 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:15.950522 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:15.972512 (+ 21990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:15.972517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:15.972547 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:15.972988 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.973000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.975121 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.975129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.977232 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:15.977242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.979561 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.979567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:15.979746 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.979789 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:49:15.979896 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:49:15.980041 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":1,"thread_start_us":917,"threads_started":1}
I20260928 15:49:15.981503 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:15.949269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.13.5.67:39967)
0928 15:49:15.971852 (+ 22583us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:15.971905 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:15.971942 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:49:15.972039 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:15.972047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:15.972289 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:15.973193 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:15.973446 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.974969 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.977995 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:15.979355 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:15.980618 (+ 1263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:15.980870 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:15.980945 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 15:49:15.981105 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22404,"thread_start_us":22242,"threads_started":1}
I20260928 15:49:15.982481 32383 heartbeater.cc:503] Master 127.13.5.126:33237 was elected leader, sending a full tablet report...
I20260928 15:49:15.983816 32316 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3812e7d497475b8f3d8badcd30c0d9 (DEFAULT_TABLE table=test-table [id=da91dd6077d24b098650c8d3c0469e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:15.983816 32450 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3812e7d497475b8f3d8badcd30c0d9 (DEFAULT_TABLE table=test-table [id=da91dd6077d24b098650c8d3c0469e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:15.984681 32582 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3812e7d497475b8f3d8badcd30c0d9 (DEFAULT_TABLE table=test-table [id=da91dd6077d24b098650c8d3c0469e24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:15.985314 32450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3812e7d497475b8f3d8badcd30c0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:15.985311 32316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3812e7d497475b8f3d8badcd30c0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:15.985858 32582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3812e7d497475b8f3d8badcd30c0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:15.997464 32667 tablet_bootstrap.cc:492] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: Bootstrap starting.
I20260928 15:49:16.000907 32667 tablet_bootstrap.cc:654] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:16.001477 32669 tablet_bootstrap.cc:492] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c: Bootstrap starting.
I20260928 15:49:16.003252 32668 tablet_bootstrap.cc:492] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: Bootstrap starting.
I20260928 15:49:16.003747 32667 log.cc:824] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:16.005023 32669 tablet_bootstrap.cc:654] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:16.006248 32669 log.cc:824] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:16.006747 32668 tablet_bootstrap.cc:654] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:16.007212 32667 tablet_bootstrap.cc:492] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: No bootstrap required, opened a new log
I20260928 15:49:16.007529 32667 ts_tablet_manager.cc:1397] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260928 15:49:16.008025 32668 log.cc:824] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:16.009723 32669 tablet_bootstrap.cc:492] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c: No bootstrap required, opened a new log
I20260928 15:49:16.010020 32669 ts_tablet_manager.cc:1397] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:49:16.010610 32668 tablet_bootstrap.cc:492] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: No bootstrap required, opened a new log
I20260928 15:49:16.010905 32668 ts_tablet_manager.cc:1397] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 15:49:16.018074 32668 raft_consensus.cc:348] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.018450 32668 raft_consensus.cc:374] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:16.018590 32668 raft_consensus.cc:729] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d6300ea6084093a67828b14321aad0, State: Initialized, Role: FOLLOWER
I20260928 15:49:16.019042 32667 raft_consensus.cc:348] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.019237 32668 consensus_queue.cc:260] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.019485 32667 raft_consensus.cc:374] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:16.019618 32667 raft_consensus.cc:729] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34bbaba0ca98411db0746820da79f684, State: Initialized, Role: FOLLOWER
I20260928 15:49:16.020277 32667 consensus_queue.cc:260] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.020555 32669 raft_consensus.cc:348] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.021111 32669 raft_consensus.cc:374] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:16.021245 32669 raft_consensus.cc:729] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826283a3feee430497a1998e68697a1c, State: Initialized, Role: FOLLOWER
I20260928 15:49:16.022003 32669 consensus_queue.cc:260] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.022791 32517 heartbeater.cc:503] Master 127.13.5.126:33237 was elected leader, sending a full tablet report...
I20260928 15:49:16.023348 32667 ts_tablet_manager.cc:1428] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260928 15:49:16.025080 32669 ts_tablet_manager.cc:1428] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260928 15:49:16.026712 32649 heartbeater.cc:503] Master 127.13.5.126:33237 was elected leader, sending a full tablet report...
I20260928 15:49:16.026777 32668 ts_tablet_manager.cc:1428] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260928 15:49:16.058871 32673 raft_consensus.cc:482] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:49:16.059163 32673 raft_consensus.cc:504] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.060985 32673 leader_election.cc:288] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826283a3feee430497a1998e68697a1c (127.13.5.65:38257), 34bbaba0ca98411db0746820da79f684 (127.13.5.66:35587)
I20260928 15:49:16.067858 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.061149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.13.5.67:51357)
0928 15:49:16.061349 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.061392 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:16.061526 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:16.062115 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:16.062121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:16.062161 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:16.062385 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:16.062398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.063730 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.063733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:16.065730 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:16.065744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.067177 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.067182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:16.067296 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:16.067321 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 15:49:16.067393 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 15:49:16.067504 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:49:16.067900 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.061459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:51357 (local address 127.13.5.65:38257)
0928 15:49:16.061679 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:16.061686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:16.061709 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:49:16.061796 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:16.061804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:16.061919 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:16.062050 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:16.062576 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.063602 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.065901 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.066949 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.067109 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:16.067214 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:16.067432 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 15:49:16.067580 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:49:16.068729 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.061706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:33349 (local address 127.13.5.66:35587)
0928 15:49:16.061922 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:16.061929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:16.062412 (+ 483us) server_negotiation.cc:407] Connection header received
0928 15:49:16.062632 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:16.062646 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:16.062736 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:16.062839 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:16.063306 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.064185 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.066360 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.067638 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.067769 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:16.067852 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:16.068335 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 15:49:16.068434 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:49:16.069084 32337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3812e7d497475b8f3d8badcd30c0d9" candidate_uuid: "10d6300ea6084093a67828b14321aad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826283a3feee430497a1998e68697a1c" is_pre_election: true
I20260928 15:49:16.069360 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.061543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:35587 (local address 127.13.5.67:33349)
0928 15:49:16.062290 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.062314 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:16.062432 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:16.062853 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:16.062856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:16.062875 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:16.063137 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:16.063143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.064327 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.064335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:16.066181 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:16.066191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.067801 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.067807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:16.068073 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:16.068111 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 15:49:16.068911 (+ 800us) client_negotiation.cc:240] Negotiation successful
0928 15:49:16.069054 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":385,"threads_started":1}
I20260928 15:49:16.069736 32337 raft_consensus.cc:2454] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10d6300ea6084093a67828b14321aad0 in term 0.
I20260928 15:49:16.070472 32535 leader_election.cc:302] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d6300ea6084093a67828b14321aad0, 826283a3feee430497a1998e68697a1c; no voters:
I20260928 15:49:16.070600 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3812e7d497475b8f3d8badcd30c0d9" candidate_uuid: "10d6300ea6084093a67828b14321aad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bbaba0ca98411db0746820da79f684" is_pre_election: true
I20260928 15:49:16.070922 32673 raft_consensus.cc:2781] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:49:16.071089 32673 raft_consensus.cc:482] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:49:16.071213 32673 raft_consensus.cc:3037] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:16.071213 32471 raft_consensus.cc:2454] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10d6300ea6084093a67828b14321aad0 in term 0.
I20260928 15:49:16.073840 32673 raft_consensus.cc:504] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.074517 32673 leader_election.cc:288] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [CANDIDATE]: Term 1 election: Requested vote from peers 826283a3feee430497a1998e68697a1c (127.13.5.65:38257), 34bbaba0ca98411db0746820da79f684 (127.13.5.66:35587)
I20260928 15:49:16.074981 32337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3812e7d497475b8f3d8badcd30c0d9" candidate_uuid: "10d6300ea6084093a67828b14321aad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826283a3feee430497a1998e68697a1c"
I20260928 15:49:16.075001 32471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3812e7d497475b8f3d8badcd30c0d9" candidate_uuid: "10d6300ea6084093a67828b14321aad0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34bbaba0ca98411db0746820da79f684"
I20260928 15:49:16.075201 32471 raft_consensus.cc:3037] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:16.075176 32337 raft_consensus.cc:3037] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:16.077544 32337 raft_consensus.cc:2454] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10d6300ea6084093a67828b14321aad0 in term 1.
I20260928 15:49:16.077575 32471 raft_consensus.cc:2454] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10d6300ea6084093a67828b14321aad0 in term 1.
I20260928 15:49:16.077937 32535 leader_election.cc:302] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d6300ea6084093a67828b14321aad0, 826283a3feee430497a1998e68697a1c; no voters:
I20260928 15:49:16.078226 32673 raft_consensus.cc:2781] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:16.078605 32673 raft_consensus.cc:686] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 1 LEADER]: Becoming Leader. State: Replica: 10d6300ea6084093a67828b14321aad0, State: Running, Role: LEADER
I20260928 15:49:16.079084 32673 consensus_queue.cc:237] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } }
I20260928 15:49:16.084115 32192 catalog_manager.cc:5495] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10d6300ea6084093a67828b14321aad0 (127.13.5.67). New cstate: current_term: 1 leader_uuid: "10d6300ea6084093a67828b14321aad0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d6300ea6084093a67828b14321aad0" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 } health_report { overall_health: UNKNOWN } } }
I20260928 15:49:16.109855 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.104073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.13.5.67:39967)
0928 15:49:16.104261 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:16.104266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:16.104696 (+ 430us) server_negotiation.cc:407] Connection header received
0928 15:49:16.104795 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:16.104799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:16.104879 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:16.105016 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:49:16.105350 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.106182 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.107600 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.108110 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.108209 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:16.108716 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:49:16.109218 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:49:16.109349 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:16.109444 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 15:49:16.109555 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:49:16.118378 32471 raft_consensus.cc:1261] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 1 FOLLOWER]: Refusing update from remote peer 10d6300ea6084093a67828b14321aad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:16.118381 32337 raft_consensus.cc:1261] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 1 FOLLOWER]: Refusing update from remote peer 10d6300ea6084093a67828b14321aad0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:16.119349 32677 consensus_queue.cc:1035] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826283a3feee430497a1998e68697a1c" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:16.119747 32673 consensus_queue.cc:1035] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34bbaba0ca98411db0746820da79f684" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 35587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:16.131569 32680 mvcc.cc:205] Tried to move back new op lower bound from 7334340837849055232 to 7334340837704114176. Current Snapshot: MvccSnapshot[applied={T|T < 7334340837849055232}]
I20260928 15:49:16.151146 32192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:16.151441 32192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:16.155169 32192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7ab296d652d471a9b271a31238c70cf: Sending DeleteTablet for 3 replicas of tablet 4d3812e7d497475b8f3d8badcd30c0d9
I20260928 15:49:16.156477 32582 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3812e7d497475b8f3d8badcd30c0d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:16 UTC) from {username='slave'} at 127.0.0.1:38818
I20260928 15:49:16.156477 32450 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3812e7d497475b8f3d8badcd30c0d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:16 UTC) from {username='slave'} at 127.0.0.1:58594
I20260928 15:49:16.156504 32316 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3812e7d497475b8f3d8badcd30c0d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:16 UTC) from {username='slave'} at 127.0.0.1:55266
I20260928 15:49:16.160130 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 32253
I20260928 15:49:16.169220 32691 tablet_replica.cc:330] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: stopping tablet replica
I20260928 15:49:16.169587 32690 tablet_replica.cc:330] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: stopping tablet replica
I20260928 15:49:16.169894 32691 raft_consensus.cc:2229] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:49:16.170022 32690 raft_consensus.cc:2229] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:16.170607 32690 raft_consensus.cc:2258] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:16.170764 32691 raft_consensus.cc:2258] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:16.173252 32690 ts_tablet_manager.cc:1905] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:16.173435 32689 tablet_replica.cc:330] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c: stopping tablet replica
I20260928 15:49:16.173890 32689 raft_consensus.cc:2229] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:16.174445 32689 raft_consensus.cc:2258] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:16.176244 32691 ts_tablet_manager.cc:1905] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:16.176580 32689 ts_tablet_manager.cc:1905] T 4d3812e7d497475b8f3d8badcd30c0d9 P 826283a3feee430497a1998e68697a1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:16.177450 32664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.161713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.13.5.65:38257)
0928 15:49:16.161943 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:16.161952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:16.169429 (+ 7477us) server_negotiation.cc:407] Connection header received
0928 15:49:16.169741 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:16.169747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:16.169865 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:16.169964 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:16.170625 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.171959 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.173272 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.173912 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.174053 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:16.175617 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:16.175646 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:16.175663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:16.176480 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:16.176541 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:16.176574 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:16.176660 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:16.176952 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 15:49:16.177058 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:49:16.178511 32378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:16.181021 32690 ts_tablet_manager.cc:1918] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:16.181250 32690 log.cc:1196] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/4d3812e7d497475b8f3d8badcd30c0d9
I20260928 15:49:16.182034 32690 ts_tablet_manager.cc:1939] T 4d3812e7d497475b8f3d8badcd30c0d9 P 34bbaba0ca98411db0746820da79f684: Deleting consensus metadata
I20260928 15:49:16.184103 32177 catalog_manager.cc:4841] TS 34bbaba0ca98411db0746820da79f684 (127.13.5.66:35587): tablet 4d3812e7d497475b8f3d8badcd30c0d9 (table test-table [id=da91dd6077d24b098650c8d3c0469e24]) successfully deleted
I20260928 15:49:16.187394 32691 ts_tablet_manager.cc:1918] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:16.187704 32691 log.cc:1196] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/4d3812e7d497475b8f3d8badcd30c0d9
I20260928 15:49:16.188383 32691 ts_tablet_manager.cc:1939] T 4d3812e7d497475b8f3d8badcd30c0d9 P 10d6300ea6084093a67828b14321aad0: Deleting consensus metadata
I20260928 15:49:16.189930 32177 catalog_manager.cc:4841] TS 10d6300ea6084093a67828b14321aad0 (127.13.5.67:39967): tablet 4d3812e7d497475b8f3d8badcd30c0d9 (table test-table [id=da91dd6077d24b098650c8d3c0469e24]) successfully deleted
I20260928 15:49:16.309403 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 32253
W20260928 15:49:16.329658 32176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.5.65:38257 (error 108)
I20260928 15:49:16.329880 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 32387
I20260928 15:49:16.331982 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.331363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55294)
0928 15:49:16.331580 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.331722 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260928 15:49:16.332293 32176 catalog_manager.cc:4584] TS 826283a3feee430497a1998e68697a1c (127.13.5.65:38257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d3812e7d497475b8f3d8badcd30c0d9: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
I20260928 15:49:16.338375 32666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.331081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.13.5.66:35587)
0928 15:49:16.331301 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:16.331307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:16.331329 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:49:16.331454 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:16.331460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:16.331575 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:16.331692 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:16.332548 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.333415 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.334856 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.335311 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.335473 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:16.336788 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:16.336798 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:16.336815 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:16.337484 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:16.337538 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:16.337559 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:16.337703 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:16.337814 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 15:49:16.337960 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:49:16.339233 32512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:16.386132 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.385542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55296)
0928 15:49:16.385805 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.385929 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 15:49:16.387354 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.386952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55306)
0928 15:49:16.387124 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.387207 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:49:16.448374 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.447722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55320)
0928 15:49:16.448042 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.448154 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 15:49:16.449667 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.449214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55324)
0928 15:49:16.449402 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.449504 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 15:49:16.452626 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 32387
I20260928 15:49:16.472247 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 32520
I20260928 15:49:16.479753 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.473099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.13.5.67:39967)
0928 15:49:16.473273 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:16.473279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:16.473373 (+ 94us) server_negotiation.cc:407] Connection header received
0928 15:49:16.473598 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:16.473604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:16.473707 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:16.473825 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:16.474259 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.475044 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.476480 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:16.476940 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:16.477089 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:16.478263 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:16.478273 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:16.478290 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:16.478941 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:16.478990 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:16.479010 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:16.479142 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:16.479281 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 15:49:16.479384 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:49:16.480870 32644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:16.530793 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.530239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55336)
0928 15:49:16.530511 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.530624 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 15:49:16.532179 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.531690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55344)
0928 15:49:16.531960 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.532033 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 15:49:16.607278 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 32520
I20260928 15:49:16.632225 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 32161
I20260928 15:49:16.633368 32223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:16.746812 32665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:16.746134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:38257 (local address 127.0.0.1:55350)
0928 15:49:16.746350 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:16.746504 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.65:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 15:49:16.746835 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 32161
2026-09-28T15:49:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2948 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:16 dist-test-slave-j968 krb5kdc[32700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:16 dist-test-slave-j968 krb5kdc[32700](info): set up 2 sockets
Sep 28 15:49:16 dist-test-slave-j968 krb5kdc[32700](info): commencing operation
krb5kdc: starting...
W20260928 15:49:17.874526 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:18 dist-test-slave-j968 krb5kdc[32700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:18.134734 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:42059
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:18.477133 32713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:18.477442 32713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:18.477535 32713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:18.484886 32713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:18.484997 32713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:18.485041 32713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:18.485114 32713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:18.485160 32713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:18.495712 32713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:42059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:18.497125 32713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:18.498571 32713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:18.510802 32718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:18.510854 32719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:18.511363 32721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:18.512109 32713 server_base.cc:1034] running on GCE node
I20260928 15:49:18.512967 32713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:18.514498 32713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:18.515756 32713 hybrid_clock.cc:648] HybridClock initialized: now 1790610558515693 us; error 50 us; skew 500 ppm
I20260928 15:49:18.516180 32713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:18 dist-test-slave-j968 krb5kdc[32700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610558, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:18.523746 32713 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:49:18.525416 32713 webserver.cc:460] Webserver started at http://127.13.5.126:35771/ using document root <none> and password file <none>
I20260928 15:49:18.526221 32713 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:18.526374 32713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:18.526755 32713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:18.530184 32713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "d7ac4fca31104e09b5d493d6dddc1bbd"
format_stamp: "Formatted at 2026-09-28 15:49:18 on dist-test-slave-j968"
server_key: "5f099c63699294b08cca4e5e4a6b79b4"
server_key_iv: "fcee98797c1342eb0a02044b66dc1a29"
server_key_version: "encryptionkey@0"
I20260928 15:49:18.531006 32713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "d7ac4fca31104e09b5d493d6dddc1bbd"
format_stamp: "Formatted at 2026-09-28 15:49:18 on dist-test-slave-j968"
server_key: "5f099c63699294b08cca4e5e4a6b79b4"
server_key_iv: "fcee98797c1342eb0a02044b66dc1a29"
server_key_version: "encryptionkey@0"
I20260928 15:49:18.536628 32713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:49:18.540244 32728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:18.541682 32713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:49:18.541877 32713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "d7ac4fca31104e09b5d493d6dddc1bbd"
format_stamp: "Formatted at 2026-09-28 15:49:18 on dist-test-slave-j968"
server_key: "5f099c63699294b08cca4e5e4a6b79b4"
server_key_iv: "fcee98797c1342eb0a02044b66dc1a29"
server_key_version: "encryptionkey@0"
I20260928 15:49:18.542086 32713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:18.563378 32713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:18.567926 32713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:18.568234 32713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:18.587783 32713 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:42059
I20260928 15:49:18.587803 313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:42059 every 8 connection(s)
I20260928 15:49:18.589257 32713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:18.593176 314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:18.593182 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 32713
I20260928 15:49:18.593401 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:18.593900 13333 external_mini_cluster.cc:1428] Setting key 7523b64943b8be9aa6e064746041539e
I20260928 15:49:18.607767 314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd: Bootstrap starting.
Sep 28 15:49:18 dist-test-slave-j968 krb5kdc[32700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:18.611362 314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:18.612794 314 log.cc:824] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:18.616259 314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd: No bootstrap required, opened a new log
I20260928 15:49:18.618467 317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:18.595993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.13.5.126:42059)
0928 15:49:18.596819 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:18.596839 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:18.596889 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:49:18.598683 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:18.598716 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:18.599246 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:18.599709 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:18.601118 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:18.603039 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:18.604512 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:18.605643 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:18.605838 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:18.609876 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:18.609886 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:18.609905 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:18.609948 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:18.613887 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:18.614696 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:18.614706 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:18.614714 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:18.614918 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:18.615277 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:18.615285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:18.615289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:18.616006 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:18.616305 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:18.617137 (+ 832us) server_negotiation.cc:299] Negotiation successful
0928 15:49:18.617452 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":329,"threads_started":1}
I20260928 15:49:18.625267 314 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ac4fca31104e09b5d493d6dddc1bbd" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42059 } }
I20260928 15:49:18.625653 314 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:18.625777 314 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ac4fca31104e09b5d493d6dddc1bbd, State: Initialized, Role: FOLLOWER
I20260928 15:49:18.626394 314 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ac4fca31104e09b5d493d6dddc1bbd" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42059 } }
I20260928 15:49:18.626628 314 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:18.626766 314 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:18.626942 314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:18.629774 314 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ac4fca31104e09b5d493d6dddc1bbd" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42059 } }
I20260928 15:49:18.630302 314 leader_election.cc:302] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7ac4fca31104e09b5d493d6dddc1bbd; no voters:
I20260928 15:49:18.630880 314 leader_election.cc:288] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:18.631036 319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:18.631739 319 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [term 1 LEADER]: Becoming Leader. State: Replica: d7ac4fca31104e09b5d493d6dddc1bbd, State: Running, Role: LEADER
I20260928 15:49:18.632357 319 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ac4fca31104e09b5d493d6dddc1bbd" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42059 } }
I20260928 15:49:18.633040 314 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:18.635457 321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7ac4fca31104e09b5d493d6dddc1bbd. Latest consensus state: current_term: 1 leader_uuid: "d7ac4fca31104e09b5d493d6dddc1bbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ac4fca31104e09b5d493d6dddc1bbd" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42059 } } }
I20260928 15:49:18.635356 320 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7ac4fca31104e09b5d493d6dddc1bbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7ac4fca31104e09b5d493d6dddc1bbd" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42059 } } }
I20260928 15:49:18.635775 320 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:18.635960 321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:18.639369 325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:18.644515 325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:18.653793 325 catalog_manager.cc:1347] Generated new cluster ID: b1ad2e61159b4ca58175c563ba32c52c
I20260928 15:49:18.653918 325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:18.673663 325 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:18.675143 325 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:18.682660 325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd: Generated new TSK 0
I20260928 15:49:18.683954 325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.5.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
Entry for principal kudu/127.13.5.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab.
I20260928 15:49:18.760277 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42059
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:19.104317 340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:19.104645 340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:19.104840 340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:19.112124 340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:19.112289 340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:19.112468 340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:49:19.124112 340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.65.keytab
--principal=kudu/127.13.5.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:19.125516 340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:19.126865 340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:19.140664 346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:19.140728 345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:19.140781 348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:19.142246 340 server_base.cc:1034] running on GCE node
I20260928 15:49:19.142890 340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:19.144017 340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:19.145234 340 hybrid_clock.cc:648] HybridClock initialized: now 1790610559145167 us; error 59 us; skew 500 ppm
I20260928 15:49:19.145684 340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:19 dist-test-slave-j968 krb5kdc[32700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610559, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:19.149215 340 init.cc:377] Logged in from keytab as kudu/127.13.5.65@KRBTEST.COM (short username kudu)
I20260928 15:49:19.150920 340 webserver.cc:460] Webserver started at http://127.13.5.65:39443/ using document root <none> and password file <none>
I20260928 15:49:19.151695 340 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:19.151837 340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:19.152242 340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:19.155328 340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "081c89b84a6d4949ba493433b1b22738"
format_stamp: "Formatted at 2026-09-28 15:49:19 on dist-test-slave-j968"
server_key: "5bb6b7be6df508d08159611f470a62d3"
server_key_iv: "a55c5417a83d55909a1dc95e0ba27188"
server_key_version: "encryptionkey@0"
I20260928 15:49:19.156190 340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "081c89b84a6d4949ba493433b1b22738"
format_stamp: "Formatted at 2026-09-28 15:49:19 on dist-test-slave-j968"
server_key: "5bb6b7be6df508d08159611f470a62d3"
server_key_iv: "a55c5417a83d55909a1dc95e0ba27188"
server_key_version: "encryptionkey@0"
I20260928 15:49:19.161513 340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:49:19.165144 355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:19.166548 340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:19.166733 340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "081c89b84a6d4949ba493433b1b22738"
format_stamp: "Formatted at 2026-09-28 15:49:19 on dist-test-slave-j968"
server_key: "5bb6b7be6df508d08159611f470a62d3"
server_key_iv: "a55c5417a83d55909a1dc95e0ba27188"
server_key_version: "encryptionkey@0"
I20260928 15:49:19.166949 340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:19.179759 340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:19.184180 340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:19.184480 340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:19.185400 340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:19.186736 340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:19.186837 340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:19.186988 340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:19.187065 340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:19.221218 340 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43623
I20260928 15:49:19.221277 470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43623 every 8 connection(s)
I20260928 15:49:19.222630 340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:49:19.230572 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 340
I20260928 15:49:19.230777 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:49:19.231403 13333 external_mini_cluster.cc:1428] Setting key 719c9d9447df22faab734b356d2048f9
Sep 28 15:49:19 dist-test-slave-j968 krb5kdc[32700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610559, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.65@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:19.246307 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:19.225381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:56281 (local address 127.13.5.126:42059)
0928 15:49:19.225829 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:19.225833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:19.226894 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 15:49:19.228873 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:19.228879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:19.228973 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:19.229108 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:19.231013 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:19.232090 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:19.234196 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:19.234681 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:19.234828 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:19.238832 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:19.238841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:19.238849 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:19.238894 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:19.242558 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:19.243673 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:19.243677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:19.243679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:19.243798 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:19.244377 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:19.244381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:19.244382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:19.244656 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:19.244765 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:19.245874 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0928 15:49:19.246056 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260928 15:49:19.248827 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:19.226003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:42059 (local address 127.13.5.65:56281)
0928 15:49:19.226726 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:19.226769 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:19.228595 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:19.229342 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:19.229350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:19.229901 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:19.230743 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:19.230774 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:19.232254 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:19.232259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:19.234045 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:19.234059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:19.234946 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:19.234956 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:19.235214 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:19.236033 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:19.236056 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:19.238600 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:19.242735 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:19.242747 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:19.242773 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:19.243540 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:19.243963 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:19.243972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:19.243979 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:19.244264 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:19.244810 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:19.244823 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:19.245339 (+ 516us) client_negotiation.cc:769] Sending connection context
0928 15:49:19.245713 (+ 374us) client_negotiation.cc:240] Negotiation successful
0928 15:49:19.247172 (+ 1459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260928 15:49:19.250623 471 heartbeater.cc:347] Connected to a master server at 127.13.5.126:42059
I20260928 15:49:19.251006 471 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:19.251788 471 heartbeater.cc:511] Master 127.13.5.126:42059 requested a full tablet report, sending...
I20260928 15:49:19.253986 32746 ts_manager.cc:194] Registered new tserver with Master: 081c89b84a6d4949ba493433b1b22738 (127.13.5.65:43623)
I20260928 15:49:19.255445 32746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.65@KRBTEST.COM'} at 127.13.5.65:56281
WARNING: no policy specified for kudu/127.13.5.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
Entry for principal kudu/127.13.5.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab.
I20260928 15:49:19.297032 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42059
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:19.656369 478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:19.656699 478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:19.656867 478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:19.664559 478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:19.664711 478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:19.664907 478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:49:19.675597 478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.66.keytab
--principal=kudu/127.13.5.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:19.677079 478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:19.678545 478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:19.692278 483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:19.692191 484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:19.692345 486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:19.693320 478 server_base.cc:1034] running on GCE node
I20260928 15:49:19.694003 478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:19.695001 478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:19.696235 478 hybrid_clock.cc:648] HybridClock initialized: now 1790610559696187 us; error 43 us; skew 500 ppm
I20260928 15:49:19.696605 478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:19 dist-test-slave-j968 krb5kdc[32700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610559, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:19.700066 478 init.cc:377] Logged in from keytab as kudu/127.13.5.66@KRBTEST.COM (short username kudu)
I20260928 15:49:19.701953 478 webserver.cc:460] Webserver started at http://127.13.5.66:35805/ using document root <none> and password file <none>
I20260928 15:49:19.702638 478 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:19.702739 478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:19.703014 478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:19.706040 478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "179344938ec442eca516fa2b39b69854"
format_stamp: "Formatted at 2026-09-28 15:49:19 on dist-test-slave-j968"
server_key: "5be482e3ff6b3a9993a9bbcf5095c3dd"
server_key_iv: "062f21fd70e7653e25d75b92f190ae01"
server_key_version: "encryptionkey@0"
I20260928 15:49:19.706781 478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "179344938ec442eca516fa2b39b69854"
format_stamp: "Formatted at 2026-09-28 15:49:19 on dist-test-slave-j968"
server_key: "5be482e3ff6b3a9993a9bbcf5095c3dd"
server_key_iv: "062f21fd70e7653e25d75b92f190ae01"
server_key_version: "encryptionkey@0"
I20260928 15:49:19.712792 478 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:49:19.716501 493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:19.717971 478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:19.718142 478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "179344938ec442eca516fa2b39b69854"
format_stamp: "Formatted at 2026-09-28 15:49:19 on dist-test-slave-j968"
server_key: "5be482e3ff6b3a9993a9bbcf5095c3dd"
server_key_iv: "062f21fd70e7653e25d75b92f190ae01"
server_key_version: "encryptionkey@0"
I20260928 15:49:19.718357 478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:19.743026 478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:19.747746 478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:19.748093 478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:19.749176 478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:19.750650 478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:19.750736 478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:19.750867 478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:19.750917 478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:19.785774 478 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:33505
I20260928 15:49:19.785868 606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:33505 every 8 connection(s)
I20260928 15:49:19.787184 478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:49:19.794541 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 478
I20260928 15:49:19.794742 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:49:19.795230 13333 external_mini_cluster.cc:1428] Setting key 71cea8c9d54110b3b98391e57abfe9f7
Sep 28 15:49:19 dist-test-slave-j968 krb5kdc[32700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610559, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.66@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:19.810138 609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:19.790092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:50413 (local address 127.13.5.126:42059)
0928 15:49:19.790569 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:19.790576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:19.791564 (+ 988us) server_negotiation.cc:407] Connection header received
0928 15:49:19.793444 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:19.793448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:19.793517 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:19.793615 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:19.795227 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:19.796273 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:19.798233 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:19.798989 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:19.799134 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:19.802713 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:19.802722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:19.802730 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:19.802780 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:19.806446 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:19.807564 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:19.807569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:19.807571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:19.807707 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:19.808273 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:19.808277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:19.808279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:19.808576 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:19.808704 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:19.809648 (+ 944us) server_negotiation.cc:299] Negotiation successful
0928 15:49:19.809828 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260928 15:49:19.811753 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:19.790671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:42059 (local address 127.13.5.66:50413)
0928 15:49:19.791398 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:19.791436 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:19.793178 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:19.793813 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:19.793820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:19.794372 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:19.795045 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:19.795064 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:19.796442 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:19.796447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:19.798085 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:19.798097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:19.799206 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:19.799211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:19.799399 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:19.800149 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:19.800172 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:19.802461 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:19.806669 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:19.806680 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:19.806703 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:19.807414 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:19.807832 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:19.807837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:19.807843 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:19.808151 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:19.808713 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:19.808724 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:19.809214 (+ 490us) client_negotiation.cc:769] Sending connection context
0928 15:49:19.809528 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 15:49:19.810128 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":291,"threads_started":1}
I20260928 15:49:19.813388 607 heartbeater.cc:347] Connected to a master server at 127.13.5.126:42059
I20260928 15:49:19.813727 607 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:19.814591 607 heartbeater.cc:511] Master 127.13.5.126:42059 requested a full tablet report, sending...
I20260928 15:49:19.816298 32746 ts_manager.cc:194] Registered new tserver with Master: 179344938ec442eca516fa2b39b69854 (127.13.5.66:33505)
I20260928 15:49:19.817171 32746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.66@KRBTEST.COM'} at 127.13.5.66:50413
WARNING: no policy specified for kudu/127.13.5.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
Entry for principal kudu/127.13.5.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab.
I20260928 15:49:19.859356 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42059
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:20.200531 613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:20.200881 613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:20.201059 613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:20.208638 613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:20.208802 613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:20.208999 613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:49:20.219220 613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.67.keytab
--principal=kudu/127.13.5.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:20.220631 613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:20.222086 613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:20.236029 618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:20.236109 621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:20.236109 619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:20.237022 613 server_base.cc:1034] running on GCE node
I20260928 15:49:20.237732 613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:20.238684 613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:20.239976 613 hybrid_clock.cc:648] HybridClock initialized: now 1790610560239937 us; error 52 us; skew 500 ppm
I20260928 15:49:20.240391 613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:20 dist-test-slave-j968 krb5kdc[32700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610560, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:20.243638 613 init.cc:377] Logged in from keytab as kudu/127.13.5.67@KRBTEST.COM (short username kudu)
I20260928 15:49:20.245278 613 webserver.cc:460] Webserver started at http://127.13.5.67:36661/ using document root <none> and password file <none>
I20260928 15:49:20.246125 613 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:20.246279 613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:20.246608 613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:20.249729 613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3"
format_stamp: "Formatted at 2026-09-28 15:49:20 on dist-test-slave-j968"
server_key: "250ab0250c08fbb7f3eb9f06a3c46f1b"
server_key_iv: "4b83dfe88073e08f4f4cd09f1fc68d4c"
server_key_version: "encryptionkey@0"
I20260928 15:49:20.250452 613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3"
format_stamp: "Formatted at 2026-09-28 15:49:20 on dist-test-slave-j968"
server_key: "250ab0250c08fbb7f3eb9f06a3c46f1b"
server_key_iv: "4b83dfe88073e08f4f4cd09f1fc68d4c"
server_key_version: "encryptionkey@0"
I20260928 15:49:20.255724 613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:49:20.258496 471 heartbeater.cc:503] Master 127.13.5.126:42059 was elected leader, sending a full tablet report...
I20260928 15:49:20.259564 628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:20.261214 613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:20.261345 613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3"
format_stamp: "Formatted at 2026-09-28 15:49:20 on dist-test-slave-j968"
server_key: "250ab0250c08fbb7f3eb9f06a3c46f1b"
server_key_iv: "4b83dfe88073e08f4f4cd09f1fc68d4c"
server_key_version: "encryptionkey@0"
I20260928 15:49:20.261547 613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:20.292760 613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:20.296855 613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:20.297107 613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:20.298020 613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:20.299377 613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:20.299453 613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:20.299535 613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:20.299587 613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:20.334733 613 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:38909
I20260928 15:49:20.334784 742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:38909 every 8 connection(s)
I20260928 15:49:20.336249 613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:49:20.346236 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 613
I20260928 15:49:20.346426 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:49:20.346907 13333 external_mini_cluster.cc:1428] Setting key 0f209a0f2622d19dd9c1b52c89ee4531
Sep 28 15:49:20 dist-test-slave-j968 krb5kdc[32700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610560, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.67@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:20.357667 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.339009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:53605 (local address 127.13.5.126:42059)
0928 15:49:20.339604 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.339609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.340701 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 15:49:20.342631 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.342637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.342748 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.342868 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:20.344491 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.345388 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.347259 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.347851 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.348029 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.351954 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:20.351961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:20.351965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:20.352027 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:20.354534 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:20.355275 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:20.355281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:20.355285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:20.355413 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:20.355846 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:20.355852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:20.355855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:20.356166 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:20.356335 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.357047 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.357272 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":233,"threads_started":1}
I20260928 15:49:20.358651 746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.339697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:42059 (local address 127.13.5.67:53605)
0928 15:49:20.340543 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:20.340590 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:20.342342 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:20.343126 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:20.343135 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:20.343670 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:20.344316 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.344334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.345546 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.345553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.347112 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.347124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.348104 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.348111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.348424 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.349314 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:49:20.349337 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:20.351722 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:20.354695 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:20.354704 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:20.354722 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:20.355147 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:20.355533 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:49:20.355538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:49:20.355541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:49:20.355731 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:49:20.356296 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:20.356306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:20.356677 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 15:49:20.356948 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 15:49:20.357508 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":352,"threads_started":1}
I20260928 15:49:20.360294 743 heartbeater.cc:347] Connected to a master server at 127.13.5.126:42059
I20260928 15:49:20.360694 743 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:20.361694 743 heartbeater.cc:511] Master 127.13.5.126:42059 requested a full tablet report, sending...
I20260928 15:49:20.363384 32746 ts_manager.cc:194] Registered new tserver with Master: a3a4f4f6cf9b4f1ea898c509b9f888c3 (127.13.5.67:38909)
I20260928 15:49:20.364249 32746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.5.67@KRBTEST.COM'} at 127.13.5.67:53605
I20260928 15:49:20.375422 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:49:20.392998 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.379741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.13.5.126:42059)
0928 15:49:20.379994 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.380002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.380464 (+ 462us) server_negotiation.cc:407] Connection header received
0928 15:49:20.380674 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.380687 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.380768 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.380876 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:20.382378 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.383132 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.384529 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.384936 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.385070 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.387560 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:20.387565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:20.387568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:20.387601 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:20.390228 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:20.390929 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:20.390941 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:20.390944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:20.391064 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:20.391443 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:20.391451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:20.391455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:20.391785 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:20.392082 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.392428 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.392678 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:49:20.397110 32746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:49:20.404985 32746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:49:20.440250 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.419009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.13.5.67:38909)
0928 15:49:20.431810 (+ 12801us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.431817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.431845 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:49:20.431994 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.431999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.432296 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.432552 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:20.433612 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.434709 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.437484 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.438818 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.438962 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.439219 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.439507 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.439739 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12641,"thread_start_us":12492,"threads_started":1}
I20260928 15:49:20.440546 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.418478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:38909 (local address 127.0.0.1:41280)
0928 15:49:20.429726 (+ 11248us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:20.429747 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:20.429855 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:20.432546 (+ 2691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:20.432550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:20.432595 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:20.433355 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.433363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.434923 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.434931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.437321 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.437336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.439044 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.439050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.439250 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.439313 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 15:49:20.439465 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 15:49:20.440010 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11125,"mutex_wait_us":86,"spinlock_wait_cycles":768,"thread_start_us":9970,"threads_started":1}
I20260928 15:49:20.440814 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.418323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:33505 (local address 127.0.0.1:52868)
0928 15:49:20.428871 (+ 10548us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:20.428932 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:20.429111 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:20.432297 (+ 3186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:20.432305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:20.432340 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:20.432811 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.432825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.434827 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.434835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.437123 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.437135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.439660 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.439672 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.440020 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.440059 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:49:20.440166 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:49:20.440347 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10281,"mutex_wait_us":149,"thread_start_us":11600,"threads_started":1}
I20260928 15:49:20.441002 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.419712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.13.5.65:43623)
0928 15:49:20.430607 (+ 10895us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.430618 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.431429 (+ 811us) server_negotiation.cc:407] Connection header received
0928 15:49:20.431732 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.431741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.432058 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.432237 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:20.433622 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.435244 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.438045 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.439165 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.439286 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.439516 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.440474 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.440664 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10710,"thread_start_us":10536,"threads_started":1}
I20260928 15:49:20.441455 760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.419055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43623 (local address 127.0.0.1:58634)
0928 15:49:20.431318 (+ 12263us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:20.431338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:20.431538 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:20.432554 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:20.432558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:20.432602 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:20.433350 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.433363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.435966 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.435972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.437878 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.437891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.439685 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.439691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.439942 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.439975 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:49:20.440094 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 15:49:20.440241 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12141,"mutex_wait_us":151,"spinlock_wait_cycles":15488,"thread_start_us":12005,"threads_started":1}
I20260928 15:49:20.442518 758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.418533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.13.5.66:33505)
0928 15:49:20.431705 (+ 13172us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.431714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.431740 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:49:20.431820 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.431826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.432058 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.432253 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:20.433009 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.434678 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.437282 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.438805 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.439050 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.439214 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.441045 (+ 1831us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.441228 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12980,"thread_start_us":12218,"threads_started":1}
I20260928 15:49:20.445376 405 tablet_service.cc:1467] Processing CreateTablet for tablet 660b20a6f7874d3fbaf565f4a2daabb1 (DEFAULT_TABLE table=test-table [id=d757e09f9967475299abdaa8c43cd75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:20.445376 677 tablet_service.cc:1467] Processing CreateTablet for tablet 660b20a6f7874d3fbaf565f4a2daabb1 (DEFAULT_TABLE table=test-table [id=d757e09f9967475299abdaa8c43cd75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:20.447016 405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b20a6f7874d3fbaf565f4a2daabb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:20.447016 677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b20a6f7874d3fbaf565f4a2daabb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:20.447050 541 tablet_service.cc:1467] Processing CreateTablet for tablet 660b20a6f7874d3fbaf565f4a2daabb1 (DEFAULT_TABLE table=test-table [id=d757e09f9967475299abdaa8c43cd75a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:20.448258 541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b20a6f7874d3fbaf565f4a2daabb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:20.459849 762 tablet_bootstrap.cc:492] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Bootstrap starting.
I20260928 15:49:20.463023 762 tablet_bootstrap.cc:654] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:20.463936 763 tablet_bootstrap.cc:492] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: Bootstrap starting.
I20260928 15:49:20.464469 762 log.cc:824] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:20.467674 764 tablet_bootstrap.cc:492] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Bootstrap starting.
I20260928 15:49:20.468639 763 tablet_bootstrap.cc:654] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:20.470196 763 log.cc:824] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:20.471118 764 tablet_bootstrap.cc:654] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:20.472411 764 log.cc:824] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:20.474097 762 tablet_bootstrap.cc:492] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: No bootstrap required, opened a new log
I20260928 15:49:20.474419 762 ts_tablet_manager.cc:1397] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260928 15:49:20.474498 763 tablet_bootstrap.cc:492] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: No bootstrap required, opened a new log
I20260928 15:49:20.474756 763 ts_tablet_manager.cc:1397] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 15:49:20.475622 764 tablet_bootstrap.cc:492] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: No bootstrap required, opened a new log
I20260928 15:49:20.475894 764 ts_tablet_manager.cc:1397] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 15:49:20.482004 762 raft_consensus.cc:348] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.482506 762 raft_consensus.cc:374] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:20.482607 762 raft_consensus.cc:729] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179344938ec442eca516fa2b39b69854, State: Initialized, Role: FOLLOWER
I20260928 15:49:20.483307 762 consensus_queue.cc:260] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.484464 607 heartbeater.cc:503] Master 127.13.5.126:42059 was elected leader, sending a full tablet report...
I20260928 15:49:20.485234 762 ts_tablet_manager.cc:1428] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260928 15:49:20.486677 764 raft_consensus.cc:348] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.486678 763 raft_consensus.cc:348] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.487146 764 raft_consensus.cc:374] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:20.487147 763 raft_consensus.cc:374] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:20.487253 764 raft_consensus.cc:729] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a4f4f6cf9b4f1ea898c509b9f888c3, State: Initialized, Role: FOLLOWER
I20260928 15:49:20.487253 763 raft_consensus.cc:729] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081c89b84a6d4949ba493433b1b22738, State: Initialized, Role: FOLLOWER
I20260928 15:49:20.488052 764 consensus_queue.cc:260] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.488054 763 consensus_queue.cc:260] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.490016 743 heartbeater.cc:503] Master 127.13.5.126:42059 was elected leader, sending a full tablet report...
I20260928 15:49:20.490444 763 ts_tablet_manager.cc:1428] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260928 15:49:20.490576 764 ts_tablet_manager.cc:1428] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Time spent starting tablet: real 0.015s user 0.001s sys 0.013s
W20260928 15:49:20.539431 608 tablet.cc:2369] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:49:20.588101 744 tablet.cc:2369] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:49:20.620848 768 raft_consensus.cc:482] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:49:20.621222 768 raft_consensus.cc:504] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.623328 768 leader_election.cc:288] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3a4f4f6cf9b4f1ea898c509b9f888c3 (127.13.5.67:38909), 081c89b84a6d4949ba493433b1b22738 (127.13.5.65:43623)
I20260928 15:49:20.630623 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.623912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:50235 (local address 127.13.5.67:38909)
0928 15:49:20.624149 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.624158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.625132 (+ 974us) server_negotiation.cc:407] Connection header received
0928 15:49:20.625358 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.625368 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.625434 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.625536 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:20.626170 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.626894 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.628852 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.629640 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.629801 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.629927 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.630193 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.630323 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:49:20.630847 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.623807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:38909 (local address 127.13.5.66:50235)
0928 15:49:20.625014 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:20.625030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:20.625212 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:20.625550 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:20.625555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:20.625577 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:20.626007 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.626017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.627030 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.627037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.628690 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.628703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.629874 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.629881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.630030 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.630055 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 15:49:20.630169 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:49:20.630313 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":29,"thread_start_us":854,"threads_started":1}
I20260928 15:49:20.631269 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.623851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:59363 (local address 127.13.5.65:43623)
0928 15:49:20.624122 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.624131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.625238 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 15:49:20.625470 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.625476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.625544 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.625654 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:20.626384 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.627175 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.629105 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.629983 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.630164 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.630282 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.630702 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.631030 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 15:49:20.631956 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.623845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43623 (local address 127.13.5.66:59363)
0928 15:49:20.625111 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:20.625127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:20.625294 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:20.625674 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:20.625682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:20.625724 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:20.626204 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.626214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.627341 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.627346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.628911 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:20.628924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.630231 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.630237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:20.630459 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.630518 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 15:49:20.631458 (+ 940us) client_negotiation.cc:240] Negotiation successful
0928 15:49:20.631607 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":34,"thread_start_us":719,"threads_started":1}
I20260928 15:49:20.631987 698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b20a6f7874d3fbaf565f4a2daabb1" candidate_uuid: "179344938ec442eca516fa2b39b69854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" is_pre_election: true
I20260928 15:49:20.632609 698 raft_consensus.cc:2454] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179344938ec442eca516fa2b39b69854 in term 0.
I20260928 15:49:20.633179 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b20a6f7874d3fbaf565f4a2daabb1" candidate_uuid: "179344938ec442eca516fa2b39b69854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081c89b84a6d4949ba493433b1b22738" is_pre_election: true
I20260928 15:49:20.633394 495 leader_election.cc:302] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179344938ec442eca516fa2b39b69854, a3a4f4f6cf9b4f1ea898c509b9f888c3; no voters:
I20260928 15:49:20.633713 425 raft_consensus.cc:2454] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 179344938ec442eca516fa2b39b69854 in term 0.
I20260928 15:49:20.633836 768 raft_consensus.cc:2781] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:49:20.633961 768 raft_consensus.cc:482] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:49:20.634030 768 raft_consensus.cc:3037] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:20.636439 768 raft_consensus.cc:504] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.637054 768 leader_election.cc:288] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [CANDIDATE]: Term 1 election: Requested vote from peers a3a4f4f6cf9b4f1ea898c509b9f888c3 (127.13.5.67:38909), 081c89b84a6d4949ba493433b1b22738 (127.13.5.65:43623)
I20260928 15:49:20.637547 698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b20a6f7874d3fbaf565f4a2daabb1" candidate_uuid: "179344938ec442eca516fa2b39b69854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3"
I20260928 15:49:20.637584 425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b20a6f7874d3fbaf565f4a2daabb1" candidate_uuid: "179344938ec442eca516fa2b39b69854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081c89b84a6d4949ba493433b1b22738"
I20260928 15:49:20.637787 698 raft_consensus.cc:3037] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:20.637789 425 raft_consensus.cc:3037] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:20.641515 425 raft_consensus.cc:2454] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179344938ec442eca516fa2b39b69854 in term 1.
I20260928 15:49:20.641527 698 raft_consensus.cc:2454] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 179344938ec442eca516fa2b39b69854 in term 1.
I20260928 15:49:20.642002 495 leader_election.cc:302] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 179344938ec442eca516fa2b39b69854, a3a4f4f6cf9b4f1ea898c509b9f888c3; no voters:
I20260928 15:49:20.642407 768 raft_consensus.cc:2781] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:20.642915 768 raft_consensus.cc:686] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 1 LEADER]: Becoming Leader. State: Replica: 179344938ec442eca516fa2b39b69854, State: Running, Role: LEADER
I20260928 15:49:20.643476 768 consensus_queue.cc:237] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } }
I20260928 15:49:20.649048 32745 catalog_manager.cc:5495] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 reported cstate change: term changed from 0 to 1, leader changed from <none> to 179344938ec442eca516fa2b39b69854 (127.13.5.66). New cstate: current_term: 1 leader_uuid: "179344938ec442eca516fa2b39b69854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "179344938ec442eca516fa2b39b69854" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 33505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 } health_report { overall_health: UNKNOWN } } }
I20260928 15:49:20.669833 758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.664230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.13.5.66:33505)
0928 15:49:20.664576 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.664584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.664605 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:49:20.664746 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.664750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.664839 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.665012 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:49:20.665438 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.666153 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.667546 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.668027 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.668161 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.668701 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:49:20.669227 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:49:20.669411 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.669462 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.669546 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:49:20.679965 425 raft_consensus.cc:1261] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 1 FOLLOWER]: Refusing update from remote peer 179344938ec442eca516fa2b39b69854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:20.679965 698 raft_consensus.cc:1261] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 1 FOLLOWER]: Refusing update from remote peer 179344938ec442eca516fa2b39b69854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:20.681267 773 consensus_queue.cc:1035] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a4f4f6cf9b4f1ea898c509b9f888c3" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:20.681733 768 consensus_queue.cc:1035] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081c89b84a6d4949ba493433b1b22738" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:20.694640 776 mvcc.cc:205] Tried to move back new op lower bound from 7334340856531525632 to 7334340856400199680. Current Snapshot: MvccSnapshot[applied={T|T < 7334340856531525632}]
I20260928 15:49:20.719470 32745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:20.719967 32745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:20.724711 32745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7ac4fca31104e09b5d493d6dddc1bbd: Sending DeleteTablet for 3 replicas of tablet 660b20a6f7874d3fbaf565f4a2daabb1
I20260928 15:49:20.725906 677 tablet_service.cc:1514] Processing DeleteTablet for tablet 660b20a6f7874d3fbaf565f4a2daabb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:20 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:41280
I20260928 15:49:20.726626 541 tablet_service.cc:1514] Processing DeleteTablet for tablet 660b20a6f7874d3fbaf565f4a2daabb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:20 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:52868
I20260928 15:49:20.727062 786 tablet_replica.cc:330] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: stopping tablet replica
I20260928 15:49:20.727509 405 tablet_service.cc:1514] Processing DeleteTablet for tablet 660b20a6f7874d3fbaf565f4a2daabb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:20 UTC) from {username='kudu', principal='kudu/127.13.5.126@KRBTEST.COM'} at 127.0.0.1:58634
I20260928 15:49:20.727593 786 raft_consensus.cc:2229] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:20.728344 786 raft_consensus.cc:2258] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:20.730727 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 340
I20260928 15:49:20.731208 786 ts_tablet_manager.cc:1905] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:20.740026 788 tablet_replica.cc:330] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: stopping tablet replica
I20260928 15:49:20.740052 786 ts_tablet_manager.cc:1918] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:20.740502 786 log.cc:1196] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/660b20a6f7874d3fbaf565f4a2daabb1
I20260928 15:49:20.740962 788 raft_consensus.cc:2229] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:20.741353 786 ts_tablet_manager.cc:1939] T 660b20a6f7874d3fbaf565f4a2daabb1 P a3a4f4f6cf9b4f1ea898c509b9f888c3: Deleting consensus metadata
I20260928 15:49:20.741714 788 raft_consensus.cc:2258] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:20.744346 788 ts_tablet_manager.cc:1905] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:20.744666 32731 catalog_manager.cc:4841] TS a3a4f4f6cf9b4f1ea898c509b9f888c3 (127.13.5.67:38909): tablet 660b20a6f7874d3fbaf565f4a2daabb1 (table test-table [id=d757e09f9967475299abdaa8c43cd75a]) successfully deleted
I20260928 15:49:20.748296 787 tablet_replica.cc:330] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: stopping tablet replica
I20260928 15:49:20.748756 787 raft_consensus.cc:2229] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:49:20.749511 787 raft_consensus.cc:2258] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:20.751986 787 ts_tablet_manager.cc:1905] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:20.753876 788 ts_tablet_manager.cc:1918] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:20.754128 788 log.cc:1196] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/660b20a6f7874d3fbaf565f4a2daabb1
I20260928 15:49:20.754762 788 ts_tablet_manager.cc:1939] T 660b20a6f7874d3fbaf565f4a2daabb1 P 081c89b84a6d4949ba493433b1b22738: Deleting consensus metadata
Sep 28 15:49:20 dist-test-slave-j968 krb5kdc[32700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.65@KRBTEST.COM
I20260928 15:49:20.756906 32733 catalog_manager.cc:4841] TS 081c89b84a6d4949ba493433b1b22738 (127.13.5.65:43623): tablet 660b20a6f7874d3fbaf565f4a2daabb1 (table test-table [id=d757e09f9967475299abdaa8c43cd75a]) successfully deleted
I20260928 15:49:20.760871 787 ts_tablet_manager.cc:1918] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:20.761093 787 log.cc:1196] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/660b20a6f7874d3fbaf565f4a2daabb1
I20260928 15:49:20.761703 787 ts_tablet_manager.cc:1939] T 660b20a6f7874d3fbaf565f4a2daabb1 P 179344938ec442eca516fa2b39b69854: Deleting consensus metadata
I20260928 15:49:20.763855 32732 catalog_manager.cc:4841] TS 179344938ec442eca516fa2b39b69854 (127.13.5.66:33505): tablet 660b20a6f7874d3fbaf565f4a2daabb1 (table test-table [id=d757e09f9967475299abdaa8c43cd75a]) successfully deleted
I20260928 15:49:20.765501 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.732331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.13.5.65:43623)
0928 15:49:20.732524 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.732531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.745827 (+ 13296us) server_negotiation.cc:407] Connection header received
0928 15:49:20.746085 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.746090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.746207 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.746289 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:20.747752 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.748896 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.750821 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.751490 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.751602 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.757377 (+ 5775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:20.757391 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:20.757410 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:20.757475 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:20.761623 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:20.762595 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:20.762605 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:20.762608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:20.762787 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:20.763293 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:20.763298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:20.763301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:20.763760 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:20.764096 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.764834 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.765101 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:49:20.766353 466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:20.866834 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 340
I20260928 15:49:20.885933 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 478
Sep 28 15:49:20 dist-test-slave-j968 krb5kdc[32700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.66@KRBTEST.COM
I20260928 15:49:20.903468 758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:20.886774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.13.5.66:33505)
0928 15:49:20.886977 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:20.886983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:20.887005 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:49:20.887157 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:20.887163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:20.887257 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:20.887370 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:20.888974 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.890304 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.891585 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:20.892341 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:20.892466 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:20.897121 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:20.897136 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:20.897153 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:20.897194 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:20.900408 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:20.901248 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:20.901255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:20.901260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:20.901406 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:20.901794 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:20.901797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:20.901799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:20.902153 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:20.902404 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:20.902741 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 15:49:20.902914 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:49:20.904289 602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:21.019970 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 478
I20260928 15:49:21.042409 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 613
Sep 28 15:49:21 dist-test-slave-j968 krb5kdc[32700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.67@KRBTEST.COM
I20260928 15:49:21.058831 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:21.043206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.13.5.67:38909)
0928 15:49:21.043390 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:21.043395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:21.043490 (+ 95us) server_negotiation.cc:407] Connection header received
0928 15:49:21.043553 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:21.043556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:21.043627 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:21.043729 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:21.045116 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:21.045986 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:21.047447 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:21.047937 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:21.048044 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:21.052919 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:21.052934 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:21.052951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:21.052992 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:21.055788 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:21.056703 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:21.056720 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:21.056723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:21.056869 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:21.057267 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:21.057273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:21.057277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:21.057646 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:21.057957 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:21.058255 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 15:49:21.058491 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:49:21.059485 738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:21.152642 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 613
I20260928 15:49:21.170218 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 32713
I20260928 15:49:21.171387 309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:21.285012 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 32713
2026-09-28T15:49:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4537 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T15:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:21Z Disabled control of system clock
I20260928 15:49:21.365365 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:42877
--webserver_interface=127.13.5.126
--webserver_port=0
--builtin_ntp_servers=127.13.5.84:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:49:21.725086 797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:21.725414 797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:21.725553 797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:21.733537 797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:21.733655 797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:21.733729 797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:21.733835 797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:21.733916 797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:21.743517 797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:42877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:21.745033 797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:21.747210 797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:21.760821 805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:21.760895 802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:21.761004 803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:21.762578 797 server_base.cc:1034] running on GCE node
I20260928 15:49:21.763480 797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:21.765154 797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:21.766464 797 hybrid_clock.cc:648] HybridClock initialized: now 1790610561766380 us; error 62 us; skew 500 ppm
I20260928 15:49:21.766884 797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:21.769758 797 webserver.cc:460] Webserver started at http://127.13.5.126:38237/ using document root <none> and password file <none>
I20260928 15:49:21.770560 797 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:21.770709 797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:21.771078 797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:21.774470 797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "12fe5397cbe640d981e665e9875f3b7c"
format_stamp: "Formatted at 2026-09-28 15:49:21 on dist-test-slave-j968"
server_key: "6c5effc78df5b8d6aad1d978c1654311"
server_key_iv: "8d14ab9c0a594146d995c9d3e0d1de39"
server_key_version: "encryptionkey@0"
I20260928 15:49:21.775370 797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "12fe5397cbe640d981e665e9875f3b7c"
format_stamp: "Formatted at 2026-09-28 15:49:21 on dist-test-slave-j968"
server_key: "6c5effc78df5b8d6aad1d978c1654311"
server_key_iv: "8d14ab9c0a594146d995c9d3e0d1de39"
server_key_version: "encryptionkey@0"
I20260928 15:49:21.781365 797 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 15:49:21.785406 811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:21.786957 797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:21.787181 797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "12fe5397cbe640d981e665e9875f3b7c"
format_stamp: "Formatted at 2026-09-28 15:49:21 on dist-test-slave-j968"
server_key: "6c5effc78df5b8d6aad1d978c1654311"
server_key_iv: "8d14ab9c0a594146d995c9d3e0d1de39"
server_key_version: "encryptionkey@0"
I20260928 15:49:21.787442 797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:21.810137 797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:21.811127 797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:21.811451 797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:21.831264 797 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:42877
I20260928 15:49:21.831274 863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:42877 every 8 connection(s)
I20260928 15:49:21.832845 797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:21.837057 864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:21.841142 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 797
I20260928 15:49:21.841336 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:21.841938 13333 external_mini_cluster.cc:1428] Setting key 4674d5eda7df92fc80fbf352eb4f693b
I20260928 15:49:21.851433 864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c: Bootstrap starting.
I20260928 15:49:21.855401 864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:21.856645 867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:21.844082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.13.5.126:42877)
0928 15:49:21.845149 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:21.845160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:21.845203 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:49:21.846863 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:21.846888 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:21.847450 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:21.847918 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:21.848583 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:21.849963 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:21.851845 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:21.852592 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:21.852730 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:21.854180 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:21.854187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:21.854201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:21.854749 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:21.854845 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:21.854866 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:21.854955 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:21.855208 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:49:21.855570 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":412,"threads_started":1}
I20260928 15:49:21.856637 864 log.cc:824] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:21.859792 864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c: No bootstrap required, opened a new log
I20260928 15:49:21.867820 864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe5397cbe640d981e665e9875f3b7c" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42877 } }
I20260928 15:49:21.868345 864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:21.868463 864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fe5397cbe640d981e665e9875f3b7c, State: Initialized, Role: FOLLOWER
I20260928 15:49:21.869262 864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe5397cbe640d981e665e9875f3b7c" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42877 } }
I20260928 15:49:21.869565 864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:21.869745 864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:21.869947 864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:21.872418 864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe5397cbe640d981e665e9875f3b7c" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42877 } }
I20260928 15:49:21.872993 864 leader_election.cc:302] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12fe5397cbe640d981e665e9875f3b7c; no voters:
I20260928 15:49:21.873654 864 leader_election.cc:288] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:21.873787 869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:21.874650 869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [term 1 LEADER]: Becoming Leader. State: Replica: 12fe5397cbe640d981e665e9875f3b7c, State: Running, Role: LEADER
I20260928 15:49:21.875192 869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe5397cbe640d981e665e9875f3b7c" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42877 } }
I20260928 15:49:21.876185 864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:21.878115 871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12fe5397cbe640d981e665e9875f3b7c. Latest consensus state: current_term: 1 leader_uuid: "12fe5397cbe640d981e665e9875f3b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe5397cbe640d981e665e9875f3b7c" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42877 } } }
I20260928 15:49:21.878477 871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:21.879267 870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12fe5397cbe640d981e665e9875f3b7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe5397cbe640d981e665e9875f3b7c" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 42877 } } }
I20260928 15:49:21.879598 870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:21.880844 874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:21.886708 874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:21.896327 874 catalog_manager.cc:1347] Generated new cluster ID: 9ed433249ed54d4e90fcdac1f0ae1826
I20260928 15:49:21.896453 874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:21.913899 874 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:21.915832 874 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:21.924384 874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c: Generated new TSK 0
I20260928 15:49:21.925539 874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:21.931128 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.65:0
--local_ip_for_outbound_sockets=127.13.5.65
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42877
--builtin_ntp_servers=127.13.5.84:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:49:22.274211 888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:22.274574 888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:22.274739 888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:22.282627 888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:22.282768 888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:22.282958 888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.65
I20260928 15:49:22.293076 888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.5.65
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.65
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:22.294464 888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:22.296398 888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:22.309865 894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:22.309988 896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:22.309901 893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:22.310762 888 server_base.cc:1034] running on GCE node
I20260928 15:49:22.311480 888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:22.312557 888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:22.313802 888 hybrid_clock.cc:648] HybridClock initialized: now 1790610562313755 us; error 42 us; skew 500 ppm
I20260928 15:49:22.314158 888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:22.316679 888 webserver.cc:460] Webserver started at http://127.13.5.65:41209/ using document root <none> and password file <none>
I20260928 15:49:22.317412 888 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:22.317554 888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:22.317881 888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:22.320931 888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/data/instance:
uuid: "6feab3b2a49648e289eb55f86237b998"
format_stamp: "Formatted at 2026-09-28 15:49:22 on dist-test-slave-j968"
server_key: "31338320f3be92073edac92b36d935fe"
server_key_iv: "35c9f499544c7ef373051ab81ed2c854"
server_key_version: "encryptionkey@0"
I20260928 15:49:22.321775 888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance:
uuid: "6feab3b2a49648e289eb55f86237b998"
format_stamp: "Formatted at 2026-09-28 15:49:22 on dist-test-slave-j968"
server_key: "31338320f3be92073edac92b36d935fe"
server_key_iv: "35c9f499544c7ef373051ab81ed2c854"
server_key_version: "encryptionkey@0"
I20260928 15:49:22.327333 888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:49:22.331264 902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:22.332759 888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:22.332957 888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/wal
uuid: "6feab3b2a49648e289eb55f86237b998"
format_stamp: "Formatted at 2026-09-28 15:49:22 on dist-test-slave-j968"
server_key: "31338320f3be92073edac92b36d935fe"
server_key_iv: "35c9f499544c7ef373051ab81ed2c854"
server_key_version: "encryptionkey@0"
I20260928 15:49:22.333212 888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:22.369620 888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:22.370639 888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:22.370936 888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:22.372043 888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:22.373641 888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:22.373766 888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:22.373911 888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:22.374006 888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:22.408481 888 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.65:43219
I20260928 15:49:22.408588 1015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.65:43219 every 8 connection(s)
I20260928 15:49:22.409835 888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/data/info.pb
I20260928 15:49:22.418140 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 888
I20260928 15:49:22.418320 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/wal/instance
I20260928 15:49:22.418853 13333 external_mini_cluster.cc:1428] Setting key 1b19a90ad994b82d14f0e3011cf31fd4
I20260928 15:49:22.421877 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.66:0
--local_ip_for_outbound_sockets=127.13.5.66
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42877
--builtin_ntp_servers=127.13.5.84:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:49:22.424127 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:22.412736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.65:60999 (local address 127.13.5.126:42877)
0928 15:49:22.413243 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:22.413248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:22.414515 (+ 1267us) server_negotiation.cc:407] Connection header received
0928 15:49:22.416396 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:22.416403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:22.416501 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:22.416621 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:22.417716 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:22.418833 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:22.420790 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:22.421397 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:22.421545 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:22.423004 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:22.423014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:22.423022 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:22.423049 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:22.423106 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:22.423125 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:22.423222 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:22.423499 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 15:49:22.423613 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":294,"threads_started":1}
I20260928 15:49:22.425181 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:22.413393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:42877 (local address 127.13.5.65:60999)
0928 15:49:22.414337 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:22.414377 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:22.416127 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:22.416857 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:22.416865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:22.416960 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:22.417547 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:22.417564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:22.419046 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:22.419050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:22.420624 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:22.420642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:22.421635 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:22.421640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:22.421853 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:22.422673 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:22.422676 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:22.422719 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:22.422801 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:22.423228 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:22.423236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:22.423242 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:49:22.423460 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 15:49:22.423858 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":292,"threads_started":1}
I20260928 15:49:22.426836 1016 heartbeater.cc:347] Connected to a master server at 127.13.5.126:42877
I20260928 15:49:22.427196 1016 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:22.428182 1016 heartbeater.cc:511] Master 127.13.5.126:42877 requested a full tablet report, sending...
I20260928 15:49:22.430645 828 ts_manager.cc:194] Registered new tserver with Master: 6feab3b2a49648e289eb55f86237b998 (127.13.5.65:43219)
I20260928 15:49:22.432705 828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.65:60999
W20260928 15:49:22.760905 1020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:22.761191 1020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:22.761276 1020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:22.768746 1020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:22.768852 1020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:22.768959 1020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.66
I20260928 15:49:22.778445 1020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.5.66
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.66
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:22.779805 1020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:22.781733 1020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:22.795078 1025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:22.795049 1026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:22.795248 1028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:22.795794 1020 server_base.cc:1034] running on GCE node
I20260928 15:49:22.796609 1020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:22.797539 1020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:22.798738 1020 hybrid_clock.cc:648] HybridClock initialized: now 1790610562798692 us; error 49 us; skew 500 ppm
I20260928 15:49:22.799039 1020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:22.801527 1020 webserver.cc:460] Webserver started at http://127.13.5.66:45221/ using document root <none> and password file <none>
I20260928 15:49:22.802191 1020 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:22.802276 1020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:22.802547 1020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:22.805500 1020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/data/instance:
uuid: "4c0c82f9966c425585b20ea2092ebfec"
format_stamp: "Formatted at 2026-09-28 15:49:22 on dist-test-slave-j968"
server_key: "44e191c67940fdecd806fdf484bbd4d1"
server_key_iv: "cbc2371509f63fc46abde5568a326c1d"
server_key_version: "encryptionkey@0"
I20260928 15:49:22.806222 1020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance:
uuid: "4c0c82f9966c425585b20ea2092ebfec"
format_stamp: "Formatted at 2026-09-28 15:49:22 on dist-test-slave-j968"
server_key: "44e191c67940fdecd806fdf484bbd4d1"
server_key_iv: "cbc2371509f63fc46abde5568a326c1d"
server_key_version: "encryptionkey@0"
I20260928 15:49:22.811367 1020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:49:22.814994 1034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:22.816780 1020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:22.816939 1020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/wal
uuid: "4c0c82f9966c425585b20ea2092ebfec"
format_stamp: "Formatted at 2026-09-28 15:49:22 on dist-test-slave-j968"
server_key: "44e191c67940fdecd806fdf484bbd4d1"
server_key_iv: "cbc2371509f63fc46abde5568a326c1d"
server_key_version: "encryptionkey@0"
I20260928 15:49:22.817143 1020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:22.837554 1020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:22.838606 1020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:22.838898 1020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:22.840165 1020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:22.842147 1020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:22.842255 1020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:22.842355 1020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:22.842423 1020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:22.877710 1020 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.66:39501
I20260928 15:49:22.877733 1146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.66:39501 every 8 connection(s)
I20260928 15:49:22.879074 1020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/data/info.pb
I20260928 15:49:22.880759 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 1020
I20260928 15:49:22.880973 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/wal/instance
I20260928 15:49:22.881599 13333 external_mini_cluster.cc:1428] Setting key 6ecbbbec536ad7c6f22cd7deae91fefb
I20260928 15:49:22.884740 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.5.67:0
--local_ip_for_outbound_sockets=127.13.5.67
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42877
--builtin_ntp_servers=127.13.5.84:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:49:22.893148 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:22.881953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:59797 (local address 127.13.5.126:42877)
0928 15:49:22.882129 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:22.882133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:22.883913 (+ 1780us) server_negotiation.cc:407] Connection header received
0928 15:49:22.885795 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:22.885802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:22.885895 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:22.886019 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:22.887043 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:22.888214 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:22.889801 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:22.890509 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:22.890650 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:22.892157 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:22.892165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:22.892173 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:22.892198 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:22.892240 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:22.892259 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:22.892350 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:22.892590 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 15:49:22.892716 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:49:22.894866 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:22.882581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:42877 (local address 127.13.5.66:59797)
0928 15:49:22.883717 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:22.883769 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:22.885518 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:22.886212 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:22.886220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:22.886307 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:22.886869 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:22.886885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:22.888376 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:22.888387 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:22.889653 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:22.889666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:22.890711 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:22.890716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:22.890974 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:22.891747 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:22.891750 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:22.891794 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:22.891916 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:22.892354 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:22.892362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:22.892368 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:49:22.892548 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 15:49:22.893342 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":302,"threads_started":1}
I20260928 15:49:22.896360 1147 heartbeater.cc:347] Connected to a master server at 127.13.5.126:42877
I20260928 15:49:22.896675 1147 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:22.897428 1147 heartbeater.cc:511] Master 127.13.5.126:42877 requested a full tablet report, sending...
I20260928 15:49:22.898908 828 ts_manager.cc:194] Registered new tserver with Master: 4c0c82f9966c425585b20ea2092ebfec (127.13.5.66:39501)
I20260928 15:49:22.900167 828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.66:59797
W20260928 15:49:23.224833 1150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:23.225126 1150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:23.225211 1150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:23.232508 1150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:23.232631 1150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:23.232743 1150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.5.67
I20260928 15:49:23.242153 1150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.5.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.5.67
--webserver_port=0
--tserver_master_addrs=127.13.5.126:42877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.5.67
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:23.243465 1150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:23.245303 1150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:23.259189 1159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:23.259229 1155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:23.259301 1156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:23.260008 1150 server_base.cc:1034] running on GCE node
I20260928 15:49:23.260568 1150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:23.261593 1150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:23.262842 1150 hybrid_clock.cc:648] HybridClock initialized: now 1790610563262796 us; error 32 us; skew 500 ppm
I20260928 15:49:23.263186 1150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:49:23.265646 1150 webserver.cc:460] Webserver started at http://127.13.5.67:33323/ using document root <none> and password file <none>
I20260928 15:49:23.266381 1150 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:23.266518 1150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:23.266855 1150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:23.269905 1150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/data/instance:
uuid: "07cf6d750f3649c4a75ac14832a827fc"
format_stamp: "Formatted at 2026-09-28 15:49:23 on dist-test-slave-j968"
server_key: "b53f4b9e2478aa019a5bdc70446181c3"
server_key_iv: "8ea8dd05c799e4167f6cd8a4aa0d3f98"
server_key_version: "encryptionkey@0"
I20260928 15:49:23.270722 1150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance:
uuid: "07cf6d750f3649c4a75ac14832a827fc"
format_stamp: "Formatted at 2026-09-28 15:49:23 on dist-test-slave-j968"
server_key: "b53f4b9e2478aa019a5bdc70446181c3"
server_key_iv: "8ea8dd05c799e4167f6cd8a4aa0d3f98"
server_key_version: "encryptionkey@0"
I20260928 15:49:23.276084 1150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:49:23.279789 1165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:23.281193 1150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:23.281400 1150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/wal
uuid: "07cf6d750f3649c4a75ac14832a827fc"
format_stamp: "Formatted at 2026-09-28 15:49:23 on dist-test-slave-j968"
server_key: "b53f4b9e2478aa019a5bdc70446181c3"
server_key_iv: "8ea8dd05c799e4167f6cd8a4aa0d3f98"
server_key_version: "encryptionkey@0"
I20260928 15:49:23.281654 1150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:23.295940 1150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:23.297086 1150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:23.297544 1150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:23.298687 1150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:49:23.300297 1150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:49:23.300419 1150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:23.300562 1150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:49:23.300660 1150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:23.335450 1150 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.67:39675
I20260928 15:49:23.335510 1279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.67:39675 every 8 connection(s)
I20260928 15:49:23.336932 1150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/data/info.pb
I20260928 15:49:23.343806 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 1150
I20260928 15:49:23.344028 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/wal/instance
I20260928 15:49:23.344596 13333 external_mini_cluster.cc:1428] Setting key 9f1561b40e52802bb071f65a6e4babe9
I20260928 15:49:23.350960 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.340013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.67:52637 (local address 127.13.5.126:42877)
0928 15:49:23.340241 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.340248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.341749 (+ 1501us) server_negotiation.cc:407] Connection header received
0928 15:49:23.343567 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.343572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.343644 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.343736 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:23.345269 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.346346 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.348372 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.348789 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.348888 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.350279 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:23.350283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:23.350286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:23.350304 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:23.350336 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:23.350348 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:23.350415 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.350643 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.350724 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:49:23.352120 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.340524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:42877 (local address 127.13.5.67:52637)
0928 15:49:23.341578 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:23.341617 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:23.343219 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:23.344085 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:23.344099 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:23.344226 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:49:23.345064 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.345091 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.346596 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.346602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.348243 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.348255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.349015 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.349020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.349234 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.349939 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:49:23.349942 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:49:23.350001 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:49:23.350083 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:49:23.350426 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:49:23.350433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:49:23.350439 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:49:23.350592 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 15:49:23.350890 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":359,"threads_started":1}
I20260928 15:49:23.353482 1280 heartbeater.cc:347] Connected to a master server at 127.13.5.126:42877
I20260928 15:49:23.353747 1280 heartbeater.cc:464] Registering TS with master...
I20260928 15:49:23.354420 1280 heartbeater.cc:511] Master 127.13.5.126:42877 requested a full tablet report, sending...
I20260928 15:49:23.355794 828 ts_manager.cc:194] Registered new tserver with Master: 07cf6d750f3649c4a75ac14832a827fc (127.13.5.67:39675)
I20260928 15:49:23.356678 828 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.5.67:52637
I20260928 15:49:23.360164 13333 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:49:23.369792 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.364249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.13.5.126:42877)
0928 15:49:23.364426 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.364431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.364888 (+ 457us) server_negotiation.cc:407] Connection header received
0928 15:49:23.365038 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.365042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.365114 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.365201 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:23.365645 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.366334 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.367655 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.368130 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.368226 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.369209 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:23.369213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:23.369217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:23.369234 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:23.369263 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:23.369275 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:23.369345 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.369478 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.369552 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:49:23.373637 828 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:49:23.380707 828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:49:23.415093 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.394260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39675 (local address 127.0.0.1:60230)
0928 15:49:23.405615 (+ 11355us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:23.405637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:23.405769 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:23.407689 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:23.407694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:23.407722 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:23.408074 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.408085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.410078 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.410083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.412295 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.412315 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.414281 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.414288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.414467 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.414524 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:49:23.414610 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:49:23.414757 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11196,"thread_start_us":11035,"threads_started":1}
I20260928 15:49:23.415269 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.394537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.13.5.67:39675)
0928 15:49:23.406361 (+ 11824us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.406371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.406408 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:49:23.406630 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.406640 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.406955 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.407176 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:23.408302 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.409931 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.412538 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.414049 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.414198 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.414486 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.414717 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.414938 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11619,"thread_start_us":12888,"threads_started":1}
I20260928 15:49:23.416698 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.393806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.13.5.66:39501)
0928 15:49:23.395619 (+ 1813us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.395631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.396052 (+ 421us) server_negotiation.cc:407] Connection header received
0928 15:49:23.396199 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.396210 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.396543 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.396788 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:23.397961 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.400112 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.404179 (+ 4067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.406194 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.406331 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.406607 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.416140 (+ 9533us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.416355 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1594,"thread_start_us":1442,"threads_started":1}
I20260928 15:49:23.417182 1292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.393465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.66:39501 (local address 127.0.0.1:36546)
0928 15:49:23.394248 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:23.394286 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:23.394486 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:23.397104 (+ 2618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:23.397116 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:23.397159 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:23.397675 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.397693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.400334 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.400343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.403809 (+ 3466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.403836 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.406751 (+ 2915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.406757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.407006 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.407076 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 15:49:23.407197 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 15:49:23.407448 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":331,"threads_started":1}
I20260928 15:49:23.418570 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.394838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43219 (local address 127.0.0.1:35608)
0928 15:49:23.409924 (+ 15086us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:23.409941 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:23.410077 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:23.411771 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:23.411777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:23.411822 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:23.412364 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.412378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.413762 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.413768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.415792 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.415809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.417800 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.417806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.418007 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.418042 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:49:23.418151 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:49:23.418279 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14955,"thread_start_us":14791,"threads_started":1}
I20260928 15:49:23.418763 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.395141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.13.5.65:43219)
0928 15:49:23.410666 (+ 15525us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.410680 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.410717 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:49:23.410822 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.410834 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.411147 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.411344 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:23.412588 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.413605 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.415997 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.417525 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.417654 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.417943 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.418209 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.418372 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15281,"thread_start_us":16385,"threads_started":1}
I20260928 15:49:23.420148 1214 tablet_service.cc:1467] Processing CreateTablet for tablet d463857e4c2941f2bc13bc8c28588660 (DEFAULT_TABLE table=test-table [id=eec6578e7da0467aae2ea1c08d608bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:23.421880 1214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d463857e4c2941f2bc13bc8c28588660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:23.421932 1082 tablet_service.cc:1467] Processing CreateTablet for tablet d463857e4c2941f2bc13bc8c28588660 (DEFAULT_TABLE table=test-table [id=eec6578e7da0467aae2ea1c08d608bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:23.422555 950 tablet_service.cc:1467] Processing CreateTablet for tablet d463857e4c2941f2bc13bc8c28588660 (DEFAULT_TABLE table=test-table [id=eec6578e7da0467aae2ea1c08d608bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:49:23.423219 1082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d463857e4c2941f2bc13bc8c28588660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:23.423931 950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d463857e4c2941f2bc13bc8c28588660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:23.434085 1298 tablet_bootstrap.cc:492] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Bootstrap starting.
I20260928 15:49:23.435978 1016 heartbeater.cc:503] Master 127.13.5.126:42877 was elected leader, sending a full tablet report...
I20260928 15:49:23.438370 1298 tablet_bootstrap.cc:654] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:23.439388 1298 log.cc:824] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:23.442875 1301 tablet_bootstrap.cc:492] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Bootstrap starting.
I20260928 15:49:23.444351 1298 tablet_bootstrap.cc:492] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: No bootstrap required, opened a new log
I20260928 15:49:23.444675 1298 ts_tablet_manager.cc:1397] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260928 15:49:23.445868 1299 tablet_bootstrap.cc:492] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Bootstrap starting.
I20260928 15:49:23.447085 1301 tablet_bootstrap.cc:654] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:23.448590 1301 log.cc:824] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:23.449393 1299 tablet_bootstrap.cc:654] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:23.450567 1299 log.cc:824] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:23.452790 1301 tablet_bootstrap.cc:492] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: No bootstrap required, opened a new log
I20260928 15:49:23.453087 1301 ts_tablet_manager.cc:1397] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 15:49:23.456176 1298 raft_consensus.cc:348] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.456682 1298 raft_consensus.cc:374] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:23.456790 1298 raft_consensus.cc:729] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0c82f9966c425585b20ea2092ebfec, State: Initialized, Role: FOLLOWER
I20260928 15:49:23.456956 1299 tablet_bootstrap.cc:492] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: No bootstrap required, opened a new log
I20260928 15:49:23.457226 1299 ts_tablet_manager.cc:1397] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260928 15:49:23.457456 1298 consensus_queue.cc:260] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.464062 1301 raft_consensus.cc:348] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.464526 1301 raft_consensus.cc:374] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:23.464634 1301 raft_consensus.cc:729] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6feab3b2a49648e289eb55f86237b998, State: Initialized, Role: FOLLOWER
I20260928 15:49:23.464769 1147 heartbeater.cc:503] Master 127.13.5.126:42877 was elected leader, sending a full tablet report...
I20260928 15:49:23.465371 1298 ts_tablet_manager.cc:1428] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260928 15:49:23.465364 1301 consensus_queue.cc:260] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.469368 1299 raft_consensus.cc:348] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.469863 1299 raft_consensus.cc:374] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:23.469983 1299 raft_consensus.cc:729] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cf6d750f3649c4a75ac14832a827fc, State: Initialized, Role: FOLLOWER
I20260928 15:49:23.470661 1299 consensus_queue.cc:260] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.471808 1280 heartbeater.cc:503] Master 127.13.5.126:42877 was elected leader, sending a full tablet report...
I20260928 15:49:23.473119 1299 ts_tablet_manager.cc:1428] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260928 15:49:23.476544 1301 ts_tablet_manager.cc:1428] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Time spent starting tablet: real 0.023s user 0.017s sys 0.004s
W20260928 15:49:23.588943 1281 tablet.cc:2369] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:49:23.631284 1148 tablet.cc:2369] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:49:23.662395 1017 tablet.cc:2369] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:49:23.753602 1304 raft_consensus.cc:482] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:49:23.753990 1304 raft_consensus.cc:504] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.755821 1304 leader_election.cc:288] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cf6d750f3649c4a75ac14832a827fc (127.13.5.67:39675), 6feab3b2a49648e289eb55f86237b998 (127.13.5.65:43219)
I20260928 15:49:23.764339 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.756166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.65:43219 (local address 127.13.5.66:39187)
0928 15:49:23.757144 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:23.757163 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:23.757364 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:23.757746 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:23.757750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:23.757813 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:23.758131 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.758139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.759385 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.759389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.761674 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.761689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.763420 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.763426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.763610 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.763659 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 15:49:23.763800 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 15:49:23.763960 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"spinlock_wait_cycles":35584,"thread_start_us":689,"threads_started":1}
I20260928 15:49:23.764436 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.756488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:54335 (local address 127.13.5.67:39675)
0928 15:49:23.756706 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.756712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.757743 (+ 1031us) server_negotiation.cc:407] Connection header received
0928 15:49:23.757958 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.757962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.758030 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.758128 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:23.758744 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.759653 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.762087 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.763492 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.763625 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.763718 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.764066 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.764199 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:49:23.764653 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.756520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.5.66:39187 (local address 127.13.5.65:43219)
0928 15:49:23.756706 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.756712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.757280 (+ 568us) server_negotiation.cc:407] Connection header received
0928 15:49:23.757540 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.757545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.757617 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.757727 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:49:23.758284 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.759257 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.761926 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.763217 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.763363 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.763485 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.764220 (+ 735us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.764367 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:49:23.765046 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.756166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.67:39675 (local address 127.13.5.66:54335)
0928 15:49:23.757619 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0928 15:49:23.757634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:49:23.757810 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:49:23.758146 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:49:23.758149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:49:23.758174 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:49:23.758568 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.758582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.759785 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.759790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.761907 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:49:23.761916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.763678 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.763684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:49:23.763889 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.763920 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 15:49:23.764618 (+ 698us) client_negotiation.cc:240] Negotiation successful
0928 15:49:23.764764 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"mutex_wait_us":25,"spinlock_wait_cycles":2816,"thread_start_us":1077,"threads_started":1}
I20260928 15:49:23.765689 970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d463857e4c2941f2bc13bc8c28588660" candidate_uuid: "4c0c82f9966c425585b20ea2092ebfec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feab3b2a49648e289eb55f86237b998" is_pre_election: true
I20260928 15:49:23.766153 1234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d463857e4c2941f2bc13bc8c28588660" candidate_uuid: "4c0c82f9966c425585b20ea2092ebfec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cf6d750f3649c4a75ac14832a827fc" is_pre_election: true
I20260928 15:49:23.766350 970 raft_consensus.cc:2454] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0c82f9966c425585b20ea2092ebfec in term 0.
I20260928 15:49:23.766686 1234 raft_consensus.cc:2454] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0c82f9966c425585b20ea2092ebfec in term 0.
I20260928 15:49:23.767091 1036 leader_election.cc:302] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c82f9966c425585b20ea2092ebfec, 6feab3b2a49648e289eb55f86237b998; no voters:
I20260928 15:49:23.767539 1304 raft_consensus.cc:2781] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:49:23.767688 1304 raft_consensus.cc:482] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:49:23.767812 1304 raft_consensus.cc:3037] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:23.770117 1304 raft_consensus.cc:504] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.770780 1304 leader_election.cc:288] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [CANDIDATE]: Term 1 election: Requested vote from peers 07cf6d750f3649c4a75ac14832a827fc (127.13.5.67:39675), 6feab3b2a49648e289eb55f86237b998 (127.13.5.65:43219)
I20260928 15:49:23.771247 1234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d463857e4c2941f2bc13bc8c28588660" candidate_uuid: "4c0c82f9966c425585b20ea2092ebfec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cf6d750f3649c4a75ac14832a827fc"
I20260928 15:49:23.771348 970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d463857e4c2941f2bc13bc8c28588660" candidate_uuid: "4c0c82f9966c425585b20ea2092ebfec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feab3b2a49648e289eb55f86237b998"
I20260928 15:49:23.771426 1234 raft_consensus.cc:3037] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:23.771620 970 raft_consensus.cc:3037] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:23.773880 1234 raft_consensus.cc:2454] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0c82f9966c425585b20ea2092ebfec in term 1.
I20260928 15:49:23.773871 970 raft_consensus.cc:2454] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0c82f9966c425585b20ea2092ebfec in term 1.
I20260928 15:49:23.774333 1036 leader_election.cc:302] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c82f9966c425585b20ea2092ebfec, 6feab3b2a49648e289eb55f86237b998; no voters:
I20260928 15:49:23.774734 1304 raft_consensus.cc:2781] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:23.775234 1304 raft_consensus.cc:686] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 1 LEADER]: Becoming Leader. State: Replica: 4c0c82f9966c425585b20ea2092ebfec, State: Running, Role: LEADER
I20260928 15:49:23.775768 1304 consensus_queue.cc:237] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } }
I20260928 15:49:23.781227 826 catalog_manager.cc:5495] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c0c82f9966c425585b20ea2092ebfec (127.13.5.66). New cstate: current_term: 1 leader_uuid: "4c0c82f9966c425585b20ea2092ebfec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0c82f9966c425585b20ea2092ebfec" member_type: VOTER last_known_addr { host: "127.13.5.66" port: 39501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 } health_report { overall_health: UNKNOWN } } }
I20260928 15:49:23.865846 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.860094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.13.5.66:39501)
0928 15:49:23.860350 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.860355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.860373 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:49:23.860595 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.860599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.860682 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.860816 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:49:23.861150 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.862373 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.863812 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.864288 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.864385 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.864847 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:49:23.865373 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:49:23.865505 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.865569 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.865631 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:49:23.874249 970 raft_consensus.cc:1261] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 1 FOLLOWER]: Refusing update from remote peer 4c0c82f9966c425585b20ea2092ebfec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:23.874248 1234 raft_consensus.cc:1261] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 1 FOLLOWER]: Refusing update from remote peer 4c0c82f9966c425585b20ea2092ebfec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:49:23.875270 1309 consensus_queue.cc:1035] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6feab3b2a49648e289eb55f86237b998" member_type: VOTER last_known_addr { host: "127.13.5.65" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:23.875662 1304 consensus_queue.cc:1035] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cf6d750f3649c4a75ac14832a827fc" member_type: VOTER last_known_addr { host: "127.13.5.67" port: 39675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:49:23.899925 1312 mvcc.cc:205] Tried to move back new op lower bound from 7334340869618724864 to 7334340869230264320. Current Snapshot: MvccSnapshot[applied={T|T < 7334340869618724864}]
I20260928 15:49:23.921079 826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:23.921339 826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:49:23.925143 826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12fe5397cbe640d981e665e9875f3b7c: Sending DeleteTablet for 3 replicas of tablet d463857e4c2941f2bc13bc8c28588660
I20260928 15:49:23.926424 1214 tablet_service.cc:1514] Processing DeleteTablet for tablet d463857e4c2941f2bc13bc8c28588660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:23 UTC) from {username='slave'} at 127.0.0.1:60230
I20260928 15:49:23.926496 950 tablet_service.cc:1514] Processing DeleteTablet for tablet d463857e4c2941f2bc13bc8c28588660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:23 UTC) from {username='slave'} at 127.0.0.1:35608
I20260928 15:49:23.926685 1082 tablet_service.cc:1514] Processing DeleteTablet for tablet d463857e4c2941f2bc13bc8c28588660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:49:23 UTC) from {username='slave'} at 127.0.0.1:36546
I20260928 15:49:23.929200 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 888
I20260928 15:49:23.937415 1324 tablet_replica.cc:330] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: stopping tablet replica
I20260928 15:49:23.938074 1324 raft_consensus.cc:2229] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:23.938843 1324 raft_consensus.cc:2258] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:23.939677 1323 tablet_replica.cc:330] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: stopping tablet replica
I20260928 15:49:23.940203 1323 raft_consensus.cc:2229] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:49:23.940723 1323 raft_consensus.cc:2258] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:23.940968 1324 ts_tablet_manager.cc:1905] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:23.942883 1323 ts_tablet_manager.cc:1905] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:23.948664 1325 tablet_replica.cc:330] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: stopping tablet replica
I20260928 15:49:23.949151 1325 raft_consensus.cc:2229] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:49:23.949878 1323 ts_tablet_manager.cc:1918] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:23.949939 1325 raft_consensus.cc:2258] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:49:23.950037 1323 log.cc:1196] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-2/wal/wals/d463857e4c2941f2bc13bc8c28588660
I20260928 15:49:23.950560 1323 ts_tablet_manager.cc:1939] T d463857e4c2941f2bc13bc8c28588660 P 07cf6d750f3649c4a75ac14832a827fc: Deleting consensus metadata
I20260928 15:49:23.950846 1324 ts_tablet_manager.cc:1918] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:23.951563 1324 log.cc:1196] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-0/wal/wals/d463857e4c2941f2bc13bc8c28588660
I20260928 15:49:23.952140 1324 ts_tablet_manager.cc:1939] T d463857e4c2941f2bc13bc8c28588660 P 6feab3b2a49648e289eb55f86237b998: Deleting consensus metadata
I20260928 15:49:23.952798 1325 ts_tablet_manager.cc:1905] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:49:23.952903 812 catalog_manager.cc:4841] TS 07cf6d750f3649c4a75ac14832a827fc (127.13.5.67:39675): tablet d463857e4c2941f2bc13bc8c28588660 (table test-table [id=eec6578e7da0467aae2ea1c08d608bdb]) successfully deleted
I20260928 15:49:23.953873 813 catalog_manager.cc:4841] TS 6feab3b2a49648e289eb55f86237b998 (127.13.5.65:43219): tablet d463857e4c2941f2bc13bc8c28588660 (table test-table [id=eec6578e7da0467aae2ea1c08d608bdb]) successfully deleted
I20260928 15:49:23.957685 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:23.930660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.13.5.65:43219)
0928 15:49:23.930916 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:23.930926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:23.944495 (+ 13569us) server_negotiation.cc:407] Connection header received
0928 15:49:23.944812 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:23.944822 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:23.944965 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:23.945047 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:23.945698 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.946841 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.949079 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:23.949673 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:23.949772 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:23.955950 (+ 6178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:23.955962 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:23.955981 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:23.956768 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:23.956835 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:23.956862 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:23.956980 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:23.957173 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 15:49:23.957295 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:49:23.958520 1011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:23.959041 1325 ts_tablet_manager.cc:1918] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:49:23.959255 1325 log.cc:1196] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Deleting WAL directory at /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610238672534-13333-0/minicluster-data/ts-1/wal/wals/d463857e4c2941f2bc13bc8c28588660
I20260928 15:49:23.960172 1325 ts_tablet_manager.cc:1939] T d463857e4c2941f2bc13bc8c28588660 P 4c0c82f9966c425585b20ea2092ebfec: Deleting consensus metadata
I20260928 15:49:23.963346 814 catalog_manager.cc:4841] TS 4c0c82f9966c425585b20ea2092ebfec (127.13.5.66:39501): tablet d463857e4c2941f2bc13bc8c28588660 (table test-table [id=eec6578e7da0467aae2ea1c08d608bdb]) successfully deleted
I20260928 15:49:24.077046 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 888
I20260928 15:49:24.096979 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1020
I20260928 15:49:24.104929 1293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:24.097864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.13.5.66:39501)
0928 15:49:24.098088 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:24.098095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:24.098117 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:49:24.098192 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:24.098203 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:24.098308 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:24.098422 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:24.098767 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:24.100047 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:24.101270 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:24.101929 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:24.102067 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:24.103124 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:24.103137 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:24.103157 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:24.104084 (+ 927us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:24.104150 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:24.104177 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:24.104299 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:24.104426 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 15:49:24.104530 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:49:24.105753 1142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:24.221808 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 1020
I20260928 15:49:24.241369 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1150
I20260928 15:49:24.248775 1295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:24.242109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.13.5.67:39675)
0928 15:49:24.242322 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:24.242329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:24.242576 (+ 247us) server_negotiation.cc:407] Connection header received
0928 15:49:24.242661 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:24.242671 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:24.242757 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:24.242871 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:24.243343 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:24.244131 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:24.245487 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:24.245896 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:24.246039 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:24.247134 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:24.247143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:24.247160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:24.247745 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:24.247817 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:24.247863 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:24.248028 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:24.248194 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 15:49:24.248286 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:49:24.249441 1275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:24.339573 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 1150
I20260928 15:49:24.357313 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 797
I20260928 15:49:24.358537 859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:24.470669 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 797
2026-09-28T15:49:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3188 ms)
[----------] 6 tests from EncryptionPolicyTest (22763 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:24 dist-test-slave-j968 krb5kdc[1333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:24 dist-test-slave-j968 krb5kdc[1333](info): set up 2 sockets
Sep 28 15:49:24 dist-test-slave-j968 krb5kdc[1333](info): commencing operation
krb5kdc: starting...
W20260928 15:49:25.598969 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:25 dist-test-slave-j968 krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:25.868191 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:26.212817 1346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:26.213119 1346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:26.213210 1346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:26.220723 1346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:26.220845 1346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:26.220892 1346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:26.220954 1346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:26.220990 1346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:26.231029 1346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:26.232354 1346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:26.233703 1346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:26.247149 1351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:26.247207 1352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:26.247109 1354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:26.253392 1346 server_base.cc:1034] running on GCE node
I20260928 15:49:26.254390 1346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:26.256150 1346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:26.257486 1346 hybrid_clock.cc:648] HybridClock initialized: now 1790610566257338 us; error 90 us; skew 500 ppm
I20260928 15:49:26.257910 1346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:26 dist-test-slave-j968 krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:26.261567 1346 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:26.263305 1346 webserver.cc:460] Webserver started at http://127.0.0.1:38873/ using document root <none> and password file <none>
I20260928 15:49:26.264130 1346 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:26.264246 1346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:26.264536 1346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:26.267846 1346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "3be3ba599ca045929b102bff9d5c377e"
format_stamp: "Formatted at 2026-09-28 15:49:26 on dist-test-slave-j968"
server_key: "bb762e5a89354bd025c934d188b6a3c4"
server_key_iv: "8c90d4dfc7f5966db3d3b2e416765d0f"
server_key_version: "encryptionkey@0"
I20260928 15:49:26.268810 1346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "3be3ba599ca045929b102bff9d5c377e"
format_stamp: "Formatted at 2026-09-28 15:49:26 on dist-test-slave-j968"
server_key: "bb762e5a89354bd025c934d188b6a3c4"
server_key_iv: "8c90d4dfc7f5966db3d3b2e416765d0f"
server_key_version: "encryptionkey@0"
I20260928 15:49:26.274735 1346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:49:26.278610 1361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:26.280051 1346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:26.280268 1346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "3be3ba599ca045929b102bff9d5c377e"
format_stamp: "Formatted at 2026-09-28 15:49:26 on dist-test-slave-j968"
server_key: "bb762e5a89354bd025c934d188b6a3c4"
server_key_iv: "8c90d4dfc7f5966db3d3b2e416765d0f"
server_key_version: "encryptionkey@0"
I20260928 15:49:26.280509 1346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:26.298274 1346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:26.303049 1346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:26.303501 1346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:26.323625 1346 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36553
I20260928 15:49:26.323671 1412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36553 every 8 connection(s)
I20260928 15:49:26.325232 1346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:26.326033 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 1346
I20260928 15:49:26.326288 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:26.327075 13333 external_mini_cluster.cc:1428] Setting key 915c0470a31f61fa0fe31efba29c89ee
I20260928 15:49:26.330247 1413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:49:26 dist-test-slave-j968 krb5kdc[1333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:49:26.342054 1413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e: Bootstrap starting.
I20260928 15:49:26.345788 1413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:26.347216 1413 log.cc:824] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:26.350800 1413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e: No bootstrap required, opened a new log
I20260928 15:49:26.351095 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:26.329274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49633 (local address 127.0.0.1:36553)
0928 15:49:26.330109 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:26.330121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:26.330161 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:49:26.331851 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:26.331948 (+ 97us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:26.332341 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:26.332768 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:26.334279 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:26.335690 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:26.337058 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:26.337760 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:26.337982 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:26.342642 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:26.342650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:26.342666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:26.342706 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:26.345841 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:26.346994 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:26.347000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:26.347007 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:26.347195 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:26.347756 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:26.347760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:26.347762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:26.348746 (+ 984us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:26.349031 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:26.349637 (+ 606us) server_negotiation.cc:299] Negotiation successful
0928 15:49:26.350017 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":413,"threads_started":1}
I20260928 15:49:26.355641 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:26.361726 1413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be3ba599ca045929b102bff9d5c377e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260928 15:49:26.362162 1413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:26.362293 1413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be3ba599ca045929b102bff9d5c377e, State: Initialized, Role: FOLLOWER
I20260928 15:49:26.362959 1413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be3ba599ca045929b102bff9d5c377e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260928 15:49:26.363202 1413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:26.363335 1413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:26.363530 1413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:26.366405 1413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be3ba599ca045929b102bff9d5c377e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260928 15:49:26.367023 1413 leader_election.cc:302] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be3ba599ca045929b102bff9d5c377e; no voters:
I20260928 15:49:26.367734 1413 leader_election.cc:288] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:26.367910 1423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:26.368763 1423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [term 1 LEADER]: Becoming Leader. State: Replica: 3be3ba599ca045929b102bff9d5c377e, State: Running, Role: LEADER
I20260928 15:49:26.369519 1423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be3ba599ca045929b102bff9d5c377e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260928 15:49:26.370424 1413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:26.371829 1425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be3ba599ca045929b102bff9d5c377e. Latest consensus state: current_term: 1 leader_uuid: "3be3ba599ca045929b102bff9d5c377e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be3ba599ca045929b102bff9d5c377e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260928 15:49:26.372278 1425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:26.375371 1424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be3ba599ca045929b102bff9d5c377e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be3ba599ca045929b102bff9d5c377e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260928 15:49:26.375799 1424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:26.378417 1428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:26.381336 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:26.361693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46161 (local address 127.0.0.1:36553)
0928 15:49:26.361913 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:26.361921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:26.362220 (+ 299us) server_negotiation.cc:407] Connection header received
0928 15:49:26.362436 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:26.362441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:26.362519 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:26.362633 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:26.364345 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:26.365537 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:26.366853 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:26.367548 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:26.367706 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:26.370953 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:26.370960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:26.370971 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:26.371009 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:26.375742 (+ 4733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:26.377246 (+ 1504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:26.377254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:26.377260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:26.377554 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:26.378318 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:26.378326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:26.378332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:26.378858 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:26.380256 (+ 1398us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:26.380484 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 15:49:26.380726 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:49:26.384814 1428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:26.395151 1428 catalog_manager.cc:1347] Generated new cluster ID: f102f27478a441b4af1d1ff61220b4c2
I20260928 15:49:26.395284 1428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:26.409024 1428 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:26.410310 1428 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:26.423384 1428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be3ba599ca045929b102bff9d5c377e: Generated new TSK 0
I20260928 15:49:26.424445 1428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:26.433245 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1346
I20260928 15:49:26.434784 1408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:26.527246 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 1346
W20260928 15:49:26.545625 1331 connection.cc:535] client connection to 127.0.0.1:36553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36553): Connection reset by peer (error 104)
2026-09-28T15:49:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2047 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:26 dist-test-slave-j968 krb5kdc[1445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:26 dist-test-slave-j968 krb5kdc[1445](info): set up 2 sockets
Sep 28 15:49:26 dist-test-slave-j968 krb5kdc[1445](info): commencing operation
krb5kdc: starting...
W20260928 15:49:27.638680 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:27 dist-test-slave-j968 krb5kdc[1445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:27.911231 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:28.253088 1458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:28.253479 1458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:28.253644 1458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:28.261080 1458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:28.261250 1458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:28.261358 1458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:28.261489 1458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:28.261567 1458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:28.271760 1458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:28.273145 1458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:28.274533 1458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:28.286751 1464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:28.286839 1463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:28.287595 1467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:28.287586 1458 server_base.cc:1034] running on GCE node
I20260928 15:49:28.288516 1458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:28.290067 1458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:28.291290 1458 hybrid_clock.cc:648] HybridClock initialized: now 1790610568291221 us; error 54 us; skew 500 ppm
I20260928 15:49:28.291638 1458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:28 dist-test-slave-j968 krb5kdc[1445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:28.295184 1458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:28.296984 1458 webserver.cc:460] Webserver started at http://127.0.0.1:34791/ using document root <none> and password file <none>
I20260928 15:49:28.298022 1458 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:28.298179 1458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:28.298578 1458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:28.301812 1458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a"
format_stamp: "Formatted at 2026-09-28 15:49:28 on dist-test-slave-j968"
server_key: "5b954301395cf25d22c4fb2813c49918"
server_key_iv: "968c733bad735c7d2523434bce80f41b"
server_key_version: "encryptionkey@0"
I20260928 15:49:28.302649 1458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a"
format_stamp: "Formatted at 2026-09-28 15:49:28 on dist-test-slave-j968"
server_key: "5b954301395cf25d22c4fb2813c49918"
server_key_iv: "968c733bad735c7d2523434bce80f41b"
server_key_version: "encryptionkey@0"
I20260928 15:49:28.309242 1458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260928 15:49:28.313097 1474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:28.314565 1458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260928 15:49:28.314764 1458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a"
format_stamp: "Formatted at 2026-09-28 15:49:28 on dist-test-slave-j968"
server_key: "5b954301395cf25d22c4fb2813c49918"
server_key_iv: "968c733bad735c7d2523434bce80f41b"
server_key_version: "encryptionkey@0"
I20260928 15:49:28.314956 1458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:28.327171 1458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:28.331478 1458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:28.331771 1458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:28.351431 1458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38369
I20260928 15:49:28.351440 1525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38369 every 8 connection(s)
I20260928 15:49:28.352945 1458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:28.356972 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 1458
I20260928 15:49:28.357049 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:28.357187 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:28.357733 13333 external_mini_cluster.cc:1428] Setting key 71bf692b1376d87708eed10239eeb332
I20260928 15:49:28.373039 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:28.359691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45313 (local address 127.0.0.1:38369)
0928 15:49:28.360584 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:28.360600 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:28.360659 (+ 59us) server_negotiation.cc:407] Connection header received
0928 15:49:28.362440 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:28.362469 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:28.363009 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:28.363499 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:28.364071 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:28.366089 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:28.367430 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:28.368531 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:28.368711 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:28.370058 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:28.370069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:28.370089 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:28.370902 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:28.371004 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:28.371054 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:28.371206 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:28.371480 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 15:49:28.371669 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":398,"threads_started":1}
I20260928 15:49:28.373157 1526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a: Bootstrap starting.
I20260928 15:49:28.376520 1526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:28.377151 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:28.377849 1526 log.cc:824] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:28.381307 1526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a: No bootstrap required, opened a new log
I20260928 15:49:28.389413 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:28.382211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43115 (local address 127.0.0.1:38369)
0928 15:49:28.382441 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:28.382448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:28.382933 (+ 485us) server_negotiation.cc:407] Connection header received
0928 15:49:28.383142 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:28.383146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:28.383207 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:28.383292 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:28.383974 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:28.384773 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:28.386681 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:28.387198 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:28.387300 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:28.388662 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:28.388667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:28.388671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:28.388691 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:28.388729 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:28.388744 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:28.388862 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:28.389078 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 15:49:28.389139 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:49:28.391611 1526 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } }
I20260928 15:49:28.392072 1526 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:28.392181 1526 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36b7aa1b2cf404a9acb10eb13ddb36a, State: Initialized, Role: FOLLOWER
I20260928 15:49:28.392791 1526 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } }
I20260928 15:49:28.393030 1526 raft_consensus.cc:388] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:28.393195 1526 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:28.393373 1526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:28.395756 1526 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } }
I20260928 15:49:28.396387 1526 leader_election.cc:302] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d36b7aa1b2cf404a9acb10eb13ddb36a; no voters:
I20260928 15:49:28.397065 1526 leader_election.cc:288] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:28.397212 1536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:28.399315 1526 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:28.399734 1536 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [term 1 LEADER]: Becoming Leader. State: Replica: d36b7aa1b2cf404a9acb10eb13ddb36a, State: Running, Role: LEADER
I20260928 15:49:28.400313 1536 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } }
I20260928 15:49:28.403322 1538 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36b7aa1b2cf404a9acb10eb13ddb36a. Latest consensus state: current_term: 1 leader_uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } } }
I20260928 15:49:28.403734 1538 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:28.404309 1537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36b7aa1b2cf404a9acb10eb13ddb36a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38369 } } }
I20260928 15:49:28.404639 1537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:28.407521 1546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:28.412849 1546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:28.422797 1546 catalog_manager.cc:1347] Generated new cluster ID: 92618e50e49049b589f4dd487080a43e
I20260928 15:49:28.422925 1546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:28.432430 1546 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:28.433753 1546 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:28.445031 1546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d36b7aa1b2cf404a9acb10eb13ddb36a: Generated new TSK 0
I20260928 15:49:28.445957 1546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:28.453732 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1458
W20260928 15:49:28.455060 1521 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45313
W20260928 15:49:28.455981 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1458: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:28.456175 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 1458
W20260928 15:49:28.473830 1443 connection.cc:535] client connection to 127.0.0.1:38369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38369): Connection reset by peer (error 104)
2026-09-28T15:49:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:28 dist-test-slave-j968 krb5kdc[1557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:28 dist-test-slave-j968 krb5kdc[1557](info): set up 2 sockets
Sep 28 15:49:28 dist-test-slave-j968 krb5kdc[1557](info): commencing operation
krb5kdc: starting...
W20260928 15:49:29.602890 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:29 dist-test-slave-j968 krb5kdc[1557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:29.876643 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:30.218438 1570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:30.218819 1570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:30.218989 1570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:30.226481 1570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:30.226653 1570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:30.226771 1570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:30.226879 1570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:30.226954 1570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:30.237200 1570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:30.238586 1570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:30.239990 1570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:30.252615 1575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:30.252514 1576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:30.252583 1578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:30.253331 1570 server_base.cc:1034] running on GCE node
I20260928 15:49:30.254133 1570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:30.255698 1570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:30.256934 1570 hybrid_clock.cc:648] HybridClock initialized: now 1790610570256869 us; error 59 us; skew 500 ppm
I20260928 15:49:30.257303 1570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:30 dist-test-slave-j968 krb5kdc[1557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:30.267220 1570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:30.269044 1570 webserver.cc:460] Webserver started at http://127.0.0.1:43525/ using document root <none> and password file <none>
I20260928 15:49:30.269778 1570 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:30.269930 1570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:30.270255 1570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:30.273460 1570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "07bca754ae1446069810fbf5b108d83a"
format_stamp: "Formatted at 2026-09-28 15:49:30 on dist-test-slave-j968"
server_key: "1601d8ec1935f68bc8db3fa4aab3d8e6"
server_key_iv: "c774721420c1b78bc40fe7ff8f63eb40"
server_key_version: "encryptionkey@0"
I20260928 15:49:30.274241 1570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "07bca754ae1446069810fbf5b108d83a"
format_stamp: "Formatted at 2026-09-28 15:49:30 on dist-test-slave-j968"
server_key: "1601d8ec1935f68bc8db3fa4aab3d8e6"
server_key_iv: "c774721420c1b78bc40fe7ff8f63eb40"
server_key_version: "encryptionkey@0"
I20260928 15:49:30.279623 1570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:49:30.283365 1585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:30.284940 1570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:49:30.285135 1570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "07bca754ae1446069810fbf5b108d83a"
format_stamp: "Formatted at 2026-09-28 15:49:30 on dist-test-slave-j968"
server_key: "1601d8ec1935f68bc8db3fa4aab3d8e6"
server_key_iv: "c774721420c1b78bc40fe7ff8f63eb40"
server_key_version: "encryptionkey@0"
I20260928 15:49:30.285341 1570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:30.291256 1570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:30.295398 1570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:30.295706 1570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:30.314967 1570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34931
I20260928 15:49:30.315054 1637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34931 every 8 connection(s)
I20260928 15:49:30.316526 1570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:30.320315 1638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:30.322436 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 1570
I20260928 15:49:30.322644 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:30.323089 13333 external_mini_cluster.cc:1428] Setting key 3c2bf2c6331fdca1e2f1158e8099f2cc
I20260928 15:49:30.331248 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:30.325001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33025 (local address 127.0.0.1:34931)
0928 15:49:30.325841 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:30.325854 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:30.325900 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:49:30.327519 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:30.327542 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:30.328015 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:30.328387 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:30.329537 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:30.329544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:30.329565 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:30.329581 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:30.329641 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:30.329666 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:30.329834 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:30.330004 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 15:49:30.330235 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":391,"threads_started":1}
I20260928 15:49:30.335547 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:30.335582 1638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a: Bootstrap starting.
I20260928 15:49:30.338747 1638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:30.340024 1638 log.cc:824] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:30.342861 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:30.340178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35825 (local address 127.0.0.1:34931)
0928 15:49:30.340358 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:30.340364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:30.340782 (+ 418us) server_negotiation.cc:407] Connection header received
0928 15:49:30.341011 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:30.341016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:30.341066 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:30.341187 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:30.342149 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:30.342155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:30.342159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:30.342178 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:30.342232 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:30.342248 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:30.342371 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:30.342512 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 15:49:30.342578 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:49:30.343318 1638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a: No bootstrap required, opened a new log
I20260928 15:49:30.351249 1638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bca754ae1446069810fbf5b108d83a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260928 15:49:30.351701 1638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:30.351815 1638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bca754ae1446069810fbf5b108d83a, State: Initialized, Role: FOLLOWER
I20260928 15:49:30.352520 1638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bca754ae1446069810fbf5b108d83a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260928 15:49:30.352780 1638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:30.352928 1638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:30.353112 1638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:30.355628 1638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bca754ae1446069810fbf5b108d83a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260928 15:49:30.356220 1638 leader_election.cc:302] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07bca754ae1446069810fbf5b108d83a; no voters:
I20260928 15:49:30.356956 1638 leader_election.cc:288] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:30.357101 1648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:30.357831 1648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [term 1 LEADER]: Becoming Leader. State: Replica: 07bca754ae1446069810fbf5b108d83a, State: Running, Role: LEADER
I20260928 15:49:30.358347 1648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bca754ae1446069810fbf5b108d83a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260928 15:49:30.359169 1638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:30.361948 1650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bca754ae1446069810fbf5b108d83a. Latest consensus state: current_term: 1 leader_uuid: "07bca754ae1446069810fbf5b108d83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bca754ae1446069810fbf5b108d83a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } } }
I20260928 15:49:30.362071 1649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07bca754ae1446069810fbf5b108d83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bca754ae1446069810fbf5b108d83a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } } }
I20260928 15:49:30.362514 1649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:30.362386 1650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:30.365151 1656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:30.370759 1656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:30.380206 1656 catalog_manager.cc:1347] Generated new cluster ID: 3373666b17954aad9e0870a563328ef4
I20260928 15:49:30.380347 1656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:30.401624 1656 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:30.402115 1656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:30.409056 1656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07bca754ae1446069810fbf5b108d83a: Generated new TSK 0
I20260928 15:49:30.409966 1656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:30.417939 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1570
W20260928 15:49:30.419157 1633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33025
W20260928 15:49:30.419979 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1570: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:30.420161 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 1570
2026-09-28T15:49:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:30 dist-test-slave-j968 krb5kdc[1669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:30 dist-test-slave-j968 krb5kdc[1669](info): set up 2 sockets
Sep 28 15:49:30 dist-test-slave-j968 krb5kdc[1669](info): commencing operation
krb5kdc: starting...
W20260928 15:49:31.539050 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:31 dist-test-slave-j968 krb5kdc[1669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:31.805226 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:32.149060 1682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:32.149473 1682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:32.149638 1682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:32.157287 1682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:32.157445 1682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:32.157567 1682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:32.157677 1682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:32.157755 1682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:32.169600 1682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:32.171074 1682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:32.172528 1682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:32.184619 1687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:32.184662 1688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:32.184567 1690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:32.185503 1682 server_base.cc:1034] running on GCE node
I20260928 15:49:32.186317 1682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:32.187780 1682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:32.189037 1682 hybrid_clock.cc:648] HybridClock initialized: now 1790610572188969 us; error 56 us; skew 500 ppm
I20260928 15:49:32.189442 1682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:32 dist-test-slave-j968 krb5kdc[1669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:32.192631 1682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:32.194226 1682 webserver.cc:460] Webserver started at http://127.0.0.1:35103/ using document root <none> and password file <none>
I20260928 15:49:32.194936 1682 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:32.195087 1682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:32.195393 1682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:32.198518 1682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "aac16e33d18f42d389ac5bc2b1d61052"
format_stamp: "Formatted at 2026-09-28 15:49:32 on dist-test-slave-j968"
server_key: "ba08e2a70a738bae905c00df35cdc92f"
server_key_iv: "c9cfbb20c916bc661cfc366915ca7084"
server_key_version: "encryptionkey@0"
I20260928 15:49:32.199323 1682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "aac16e33d18f42d389ac5bc2b1d61052"
format_stamp: "Formatted at 2026-09-28 15:49:32 on dist-test-slave-j968"
server_key: "ba08e2a70a738bae905c00df35cdc92f"
server_key_iv: "c9cfbb20c916bc661cfc366915ca7084"
server_key_version: "encryptionkey@0"
I20260928 15:49:32.204738 1682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:49:32.208180 1697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:32.209494 1682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:49:32.209689 1682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "aac16e33d18f42d389ac5bc2b1d61052"
format_stamp: "Formatted at 2026-09-28 15:49:32 on dist-test-slave-j968"
server_key: "ba08e2a70a738bae905c00df35cdc92f"
server_key_iv: "c9cfbb20c916bc661cfc366915ca7084"
server_key_version: "encryptionkey@0"
I20260928 15:49:32.209888 1682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:32.226251 1682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:32.230366 1682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:32.230667 1682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:32.250061 1682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37143
I20260928 15:49:32.250090 1748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37143 every 8 connection(s)
I20260928 15:49:32.251541 1682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:32.255489 1749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:32.260555 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 1682
I20260928 15:49:32.260756 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:32.261206 13333 external_mini_cluster.cc:1428] Setting key 9022c88d2059a184ba762af51fe7e305
I20260928 15:49:32.266759 1749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052: Bootstrap starting.
I20260928 15:49:32.270617 1749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:32.271924 1749 log.cc:824] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:32.274847 1749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052: No bootstrap required, opened a new log
Sep 28 15:49:32 dist-test-slave-j968 krb5kdc[1669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:49:32.282961 1749 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac16e33d18f42d389ac5bc2b1d61052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260928 15:49:32.283383 1749 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:32.283571 1749 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac16e33d18f42d389ac5bc2b1d61052, State: Initialized, Role: FOLLOWER
I20260928 15:49:32.284332 1749 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac16e33d18f42d389ac5bc2b1d61052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260928 15:49:32.284603 1749 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:32.284765 1749 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:32.284950 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:32.263027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49761 (local address 127.0.0.1:37143)
0928 15:49:32.263957 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:32.263969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:32.264015 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:49:32.265778 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:32.265803 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:32.266212 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:32.266548 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:32.268473 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:32.269805 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:32.271622 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:32.272493 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:32.272706 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:32.276981 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:32.276992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:32.277007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:32.277048 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:32.280435 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:32.281219 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:32.281226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:32.281236 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:32.281432 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:32.281844 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:32.281851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:32.281859 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:32.282433 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:32.282752 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:32.283193 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 15:49:32.283678 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":430,"threads_started":1}
I20260928 15:49:32.284966 1749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:32.288133 1749 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac16e33d18f42d389ac5bc2b1d61052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260928 15:49:32.288774 1749 leader_election.cc:302] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac16e33d18f42d389ac5bc2b1d61052; no voters:
I20260928 15:49:32.289305 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:32.289515 1749 leader_election.cc:288] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:32.289650 1754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:32.290926 1754 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [term 1 LEADER]: Becoming Leader. State: Replica: aac16e33d18f42d389ac5bc2b1d61052, State: Running, Role: LEADER
I20260928 15:49:32.291702 1754 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac16e33d18f42d389ac5bc2b1d61052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } }
I20260928 15:49:32.292336 1749 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:32.307505 1755 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac16e33d18f42d389ac5bc2b1d61052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac16e33d18f42d389ac5bc2b1d61052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } } }
I20260928 15:49:32.308221 1755 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:32.307396 1756 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac16e33d18f42d389ac5bc2b1d61052. Latest consensus state: current_term: 1 leader_uuid: "aac16e33d18f42d389ac5bc2b1d61052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac16e33d18f42d389ac5bc2b1d61052" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37143 } } }
I20260928 15:49:32.308979 1756 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:32.319604 1769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:32.324072 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:32.295402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49135 (local address 127.0.0.1:37143)
0928 15:49:32.295622 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:32.295631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:32.305098 (+ 9467us) server_negotiation.cc:407] Connection header received
0928 15:49:32.305376 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:32.305384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:32.305519 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:32.305651 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:32.308056 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:32.309216 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:32.311166 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:32.312175 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:32.312354 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:32.316886 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:32.316897 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:32.316906 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:32.316964 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:32.321052 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:32.321868 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:32.321874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:32.321877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:32.322021 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:32.322438 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:32.322444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:32.322447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:32.322790 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:32.323063 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:32.323427 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 15:49:32.323670 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 15:49:32.325320 1769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:32.336026 1769 catalog_manager.cc:1347] Generated new cluster ID: 57d44c06eba1492a8cc400fe1db2fe6c
I20260928 15:49:32.336176 1769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:32.355582 1769 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:32.357167 1769 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:32.372227 1769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac16e33d18f42d389ac5bc2b1d61052: Generated new TSK 0
I20260928 15:49:32.373158 1769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:32.382536 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1682
I20260928 15:49:32.383647 1744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:32.478363 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 1682
2026-09-28T15:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2058 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:32 dist-test-slave-j968 krb5kdc[1782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:32 dist-test-slave-j968 krb5kdc[1782](info): set up 2 sockets
Sep 28 15:49:32 dist-test-slave-j968 krb5kdc[1782](info): commencing operation
krb5kdc: starting...
W20260928 15:49:33.598779 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:33 dist-test-slave-j968 krb5kdc[1782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:33.875293 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:34.233747 1795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:34.234064 1795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:34.234160 1795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:34.241801 1795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:34.241928 1795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:34.241977 1795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:34.242031 1795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:34.242076 1795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:34.252296 1795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:34.253809 1795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:34.255165 1795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:34.268473 1800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:34.268499 1801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:34.268549 1803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:34.269503 1795 server_base.cc:1034] running on GCE node
I20260928 15:49:34.270318 1795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:34.271999 1795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:34.273247 1795 hybrid_clock.cc:648] HybridClock initialized: now 1790610574273203 us; error 30 us; skew 500 ppm
I20260928 15:49:34.273622 1795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:34 dist-test-slave-j968 krb5kdc[1782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:34.276906 1795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:34.278345 1795 webserver.cc:460] Webserver started at http://127.0.0.1:44131/ using document root <none> and password file <none>
I20260928 15:49:34.279014 1795 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:34.279115 1795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:34.279381 1795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:34.282591 1795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "e6c492d4de684195b3c2daf4dfabdfc3"
format_stamp: "Formatted at 2026-09-28 15:49:34 on dist-test-slave-j968"
server_key: "e529f6f43ed7d64c1d5f0bf5387f8524"
server_key_iv: "cef26020d69a2a9950d051f6f2348f04"
server_key_version: "encryptionkey@0"
I20260928 15:49:34.283330 1795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "e6c492d4de684195b3c2daf4dfabdfc3"
format_stamp: "Formatted at 2026-09-28 15:49:34 on dist-test-slave-j968"
server_key: "e529f6f43ed7d64c1d5f0bf5387f8524"
server_key_iv: "cef26020d69a2a9950d051f6f2348f04"
server_key_version: "encryptionkey@0"
I20260928 15:49:34.288721 1795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:49:34.292354 1810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:34.293789 1795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:49:34.294003 1795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "e6c492d4de684195b3c2daf4dfabdfc3"
format_stamp: "Formatted at 2026-09-28 15:49:34 on dist-test-slave-j968"
server_key: "e529f6f43ed7d64c1d5f0bf5387f8524"
server_key_iv: "cef26020d69a2a9950d051f6f2348f04"
server_key_version: "encryptionkey@0"
I20260928 15:49:34.294196 1795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:34.322994 1795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:34.327553 1795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:34.327893 1795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:34.346879 1795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35751
I20260928 15:49:34.346902 1861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35751 every 8 connection(s)
I20260928 15:49:34.348451 1795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:34.352624 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 1795
I20260928 15:49:34.352830 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:34.352952 1862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:34.353502 13333 external_mini_cluster.cc:1428] Setting key cf03dcde14fdfc66377521df1255af0e
I20260928 15:49:34.368628 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:34.355589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45701 (local address 127.0.0.1:35751)
0928 15:49:34.356428 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:34.356440 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:34.356483 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:49:34.358287 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:34.358311 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:34.358767 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:34.359148 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:34.359946 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:34.361391 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:34.363489 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:34.364304 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:34.364486 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:34.365811 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:34.365821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:34.365839 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:34.366477 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:34.366569 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:34.366598 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:34.366686 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:34.367011 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 15:49:34.367330 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":408,"threads_started":1}
I20260928 15:49:34.369228 1862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3: Bootstrap starting.
I20260928 15:49:34.372701 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:34.372972 1862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:34.374322 1862 log.cc:824] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:34.378096 1862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3: No bootstrap required, opened a new log
I20260928 15:49:34.386256 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:34.378423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46073 (local address 127.0.0.1:35751)
0928 15:49:34.378608 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:34.378614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:34.379406 (+ 792us) server_negotiation.cc:407] Connection header received
0928 15:49:34.379650 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:34.379655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:34.379722 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:34.379806 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:34.380660 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:34.381415 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:34.383561 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:34.384133 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:34.384240 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:34.385507 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:34.385515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:34.385519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:34.385537 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:34.385574 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:34.385594 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:34.385664 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:34.385834 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 15:49:34.385909 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:49:34.389132 1862 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c492d4de684195b3c2daf4dfabdfc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } }
I20260928 15:49:34.389590 1862 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:34.389707 1862 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c492d4de684195b3c2daf4dfabdfc3, State: Initialized, Role: FOLLOWER
I20260928 15:49:34.390313 1862 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c492d4de684195b3c2daf4dfabdfc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } }
I20260928 15:49:34.390612 1862 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:34.390781 1862 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:34.390964 1862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:34.393519 1862 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c492d4de684195b3c2daf4dfabdfc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } }
I20260928 15:49:34.394089 1862 leader_election.cc:302] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6c492d4de684195b3c2daf4dfabdfc3; no voters:
I20260928 15:49:34.394706 1862 leader_election.cc:288] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:34.394809 1872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:34.395696 1872 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [term 1 LEADER]: Becoming Leader. State: Replica: e6c492d4de684195b3c2daf4dfabdfc3, State: Running, Role: LEADER
I20260928 15:49:34.396325 1872 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c492d4de684195b3c2daf4dfabdfc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } }
I20260928 15:49:34.397198 1862 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:34.399394 1874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6c492d4de684195b3c2daf4dfabdfc3. Latest consensus state: current_term: 1 leader_uuid: "e6c492d4de684195b3c2daf4dfabdfc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c492d4de684195b3c2daf4dfabdfc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } } }
I20260928 15:49:34.399802 1873 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6c492d4de684195b3c2daf4dfabdfc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6c492d4de684195b3c2daf4dfabdfc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35751 } } }
I20260928 15:49:34.400059 1874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:34.400135 1873 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:34.402680 1877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:34.409297 1877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:34.420084 1877 catalog_manager.cc:1347] Generated new cluster ID: 14cebb2cc31747919862e9735df527c3
I20260928 15:49:34.420231 1877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:34.452142 1877 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:34.453667 1877 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:34.468787 1877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6c492d4de684195b3c2daf4dfabdfc3: Generated new TSK 0
I20260928 15:49:34.469919 1877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:34.487784 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1795
W20260928 15:49:34.489108 1857 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45701
W20260928 15:49:34.489976 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1795: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:34.490140 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 1795
2026-09-28T15:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:34 dist-test-slave-j968 krb5kdc[1893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:34 dist-test-slave-j968 krb5kdc[1893](info): set up 2 sockets
Sep 28 15:49:34 dist-test-slave-j968 krb5kdc[1893](info): commencing operation
krb5kdc: starting...
W20260928 15:49:35.615481 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:35 dist-test-slave-j968 krb5kdc[1893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:35.895735 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:36.246949 1906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:36.247267 1906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:36.247437 1906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:36.254848 1906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:36.254971 1906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:36.255062 1906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:36.255168 1906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:36.255255 1906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:36.265513 1906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:36.266911 1906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:36.268289 1906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:36.280864 1911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:36.280893 1912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:36.280920 1914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:36.281244 1906 server_base.cc:1034] running on GCE node
I20260928 15:49:36.282339 1906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:36.283749 1906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:36.284974 1906 hybrid_clock.cc:648] HybridClock initialized: now 1790610576284897 us; error 59 us; skew 500 ppm
I20260928 15:49:36.285363 1906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:36 dist-test-slave-j968 krb5kdc[1893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:36.292928 1906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:36.294621 1906 webserver.cc:460] Webserver started at http://127.0.0.1:33241/ using document root <none> and password file <none>
I20260928 15:49:36.295342 1906 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:36.295467 1906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:36.295742 1906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:36.299139 1906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "6fb0dbd845084d7aa24f256d81840cfd"
format_stamp: "Formatted at 2026-09-28 15:49:36 on dist-test-slave-j968"
server_key: "807ee90c25224c984cf0f168e52e494a"
server_key_iv: "8be30c91ced74b9e63dd09dba3f70692"
server_key_version: "encryptionkey@0"
I20260928 15:49:36.300071 1906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "6fb0dbd845084d7aa24f256d81840cfd"
format_stamp: "Formatted at 2026-09-28 15:49:36 on dist-test-slave-j968"
server_key: "807ee90c25224c984cf0f168e52e494a"
server_key_iv: "8be30c91ced74b9e63dd09dba3f70692"
server_key_version: "encryptionkey@0"
I20260928 15:49:36.306088 1906 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:49:36.309669 1922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:36.311228 1906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:49:36.311456 1906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "6fb0dbd845084d7aa24f256d81840cfd"
format_stamp: "Formatted at 2026-09-28 15:49:36 on dist-test-slave-j968"
server_key: "807ee90c25224c984cf0f168e52e494a"
server_key_iv: "8be30c91ced74b9e63dd09dba3f70692"
server_key_version: "encryptionkey@0"
I20260928 15:49:36.311693 1906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:36.317991 1906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:36.322386 1906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:36.322712 1906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:36.342622 1906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45529
I20260928 15:49:36.342641 1973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45529 every 8 connection(s)
I20260928 15:49:36.344220 1906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:36.348088 1974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:36.352790 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 1906
I20260928 15:49:36.352999 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:36.353579 13333 external_mini_cluster.cc:1428] Setting key aa54c3260f0866b266dadb42cf046360
I20260928 15:49:36.359932 1974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd: Bootstrap starting.
I20260928 15:49:36.362088 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:36.355644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37441 (local address 127.0.0.1:45529)
0928 15:49:36.356484 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:36.356499 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:36.356548 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:49:36.358533 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:36.358556 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:36.358908 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:36.359319 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:36.360439 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:36.360445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:36.360459 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:36.360479 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:36.360535 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:36.360554 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:36.360793 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:36.360944 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 15:49:36.361133 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":379,"threads_started":1}
I20260928 15:49:36.363332 1974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:36.364511 1974 log.cc:824] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:36.366070 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:36.367435 1974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd: No bootstrap required, opened a new log
I20260928 15:49:36.373328 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:36.370658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47501 (local address 127.0.0.1:45529)
0928 15:49:36.370910 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:36.370917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:36.371289 (+ 372us) server_negotiation.cc:407] Connection header received
0928 15:49:36.371468 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:36.371473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:36.371535 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:36.371632 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:36.372611 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:36.372615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:36.372618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:36.372632 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:36.372674 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:36.372689 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:36.372781 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:36.372924 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 15:49:36.373003 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:49:36.376137 1974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb0dbd845084d7aa24f256d81840cfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260928 15:49:36.376590 1974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:36.376717 1974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb0dbd845084d7aa24f256d81840cfd, State: Initialized, Role: FOLLOWER
I20260928 15:49:36.377323 1974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb0dbd845084d7aa24f256d81840cfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260928 15:49:36.377537 1974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:36.377732 1974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:36.377875 1974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:36.380153 1974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb0dbd845084d7aa24f256d81840cfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260928 15:49:36.380623 1974 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb0dbd845084d7aa24f256d81840cfd; no voters:
I20260928 15:49:36.381250 1974 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:36.381403 1984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:36.382663 1984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [term 1 LEADER]: Becoming Leader. State: Replica: 6fb0dbd845084d7aa24f256d81840cfd, State: Running, Role: LEADER
I20260928 15:49:36.383265 1984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb0dbd845084d7aa24f256d81840cfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260928 15:49:36.383646 1974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:36.386786 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb0dbd845084d7aa24f256d81840cfd. Latest consensus state: current_term: 1 leader_uuid: "6fb0dbd845084d7aa24f256d81840cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb0dbd845084d7aa24f256d81840cfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } } }
I20260928 15:49:36.386818 1985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb0dbd845084d7aa24f256d81840cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb0dbd845084d7aa24f256d81840cfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } } }
I20260928 15:49:36.387346 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:36.387343 1985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:36.393263 1991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:36.398777 1991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:36.409126 1991 catalog_manager.cc:1347] Generated new cluster ID: 0124ac1ac4f8499894e9c3bae6da3cad
I20260928 15:49:36.409269 1991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:36.421455 1991 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:36.422065 1991 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:36.433005 1991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb0dbd845084d7aa24f256d81840cfd: Generated new TSK 0
I20260928 15:49:36.434055 1991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:36.441533 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1906
W20260928 15:49:36.442862 1969 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37441
W20260928 15:49:36.443861 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 1906: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:36.444099 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 1906
2026-09-28T15:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:36 dist-test-slave-j968 krb5kdc[2006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:36 dist-test-slave-j968 krb5kdc[2006](info): set up 2 sockets
Sep 28 15:49:36 dist-test-slave-j968 krb5kdc[2006](info): commencing operation
krb5kdc: starting...
W20260928 15:49:37.566879 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:37 dist-test-slave-j968 krb5kdc[2006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:37.839599 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:38.198239 2019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:38.198527 2019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:38.198683 2019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:38.207561 2019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:38.207676 2019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:38.207791 2019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:38.207908 2019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:38.207974 2019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:38.218449 2019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:38.219928 2019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:38.221408 2019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:38.234668 2025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:38.235121 2027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:38.235199 2024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:38.237661 2019 server_base.cc:1034] running on GCE node
I20260928 15:49:38.238483 2019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:38.240087 2019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:38.241350 2019 hybrid_clock.cc:648] HybridClock initialized: now 1790610578241308 us; error 40 us; skew 500 ppm
I20260928 15:49:38.241775 2019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:38 dist-test-slave-j968 krb5kdc[2006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:38.245136 2019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:38.246770 2019 webserver.cc:460] Webserver started at http://127.0.0.1:42317/ using document root <none> and password file <none>
I20260928 15:49:38.247510 2019 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:38.247658 2019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:38.248068 2019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:38.251360 2019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "d2fd5754477d4df28044313a93f60af4"
format_stamp: "Formatted at 2026-09-28 15:49:38 on dist-test-slave-j968"
server_key: "43f7e6b669de2a0da780c23faa31849d"
server_key_iv: "863c5decce7e6985cca52b97948ac6d3"
server_key_version: "encryptionkey@0"
I20260928 15:49:38.252346 2019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "d2fd5754477d4df28044313a93f60af4"
format_stamp: "Formatted at 2026-09-28 15:49:38 on dist-test-slave-j968"
server_key: "43f7e6b669de2a0da780c23faa31849d"
server_key_iv: "863c5decce7e6985cca52b97948ac6d3"
server_key_version: "encryptionkey@0"
I20260928 15:49:38.257906 2019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260928 15:49:38.261646 2034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:38.263232 2019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:49:38.263478 2019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "d2fd5754477d4df28044313a93f60af4"
format_stamp: "Formatted at 2026-09-28 15:49:38 on dist-test-slave-j968"
server_key: "43f7e6b669de2a0da780c23faa31849d"
server_key_iv: "863c5decce7e6985cca52b97948ac6d3"
server_key_version: "encryptionkey@0"
I20260928 15:49:38.263710 2019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:38.296622 2019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:38.302000 2019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:38.302362 2019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:38.323768 2019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46307
I20260928 15:49:38.323792 2086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46307 every 8 connection(s)
I20260928 15:49:38.325352 2019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:38.327696 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2019
I20260928 15:49:38.327951 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:38.328631 13333 external_mini_cluster.cc:1428] Setting key 69ddcc9c43f400278daae815801baeb7
I20260928 15:49:38.330132 2087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:38.344555 2087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4: Bootstrap starting.
Sep 28 15:49:38 dist-test-slave-j968 krb5kdc[2006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:49:38.348662 2087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:38.350117 2087 log.cc:824] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:38.353889 2087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4: No bootstrap required, opened a new log
I20260928 15:49:38.355907 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:38.330628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:44681 (local address 127.0.0.1:46307)
0928 15:49:38.331447 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:38.331463 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:38.331515 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:49:38.333634 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:38.333662 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:38.334216 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:38.334708 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:38.336290 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:38.338402 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:38.339991 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:38.341088 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:38.341284 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:38.346330 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:38.346342 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:38.346361 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:38.346443 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:38.350531 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:38.351758 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:38.351769 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:38.351780 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:38.352022 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:38.352550 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:38.352555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:38.352558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:38.353283 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:38.353711 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:38.354168 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 15:49:38.354550 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":327,"threads_started":1}
I20260928 15:49:38.360176 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:38.363143 2087 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd5754477d4df28044313a93f60af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260928 15:49:38.363543 2087 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:38.363672 2087 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fd5754477d4df28044313a93f60af4, State: Initialized, Role: FOLLOWER
I20260928 15:49:38.364374 2087 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd5754477d4df28044313a93f60af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260928 15:49:38.364708 2087 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:38.364890 2087 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:38.365180 2087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:38.368096 2087 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd5754477d4df28044313a93f60af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260928 15:49:38.368677 2087 leader_election.cc:302] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2fd5754477d4df28044313a93f60af4; no voters:
I20260928 15:49:38.369357 2087 leader_election.cc:288] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:38.369468 2097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:38.370791 2097 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [term 1 LEADER]: Becoming Leader. State: Replica: d2fd5754477d4df28044313a93f60af4, State: Running, Role: LEADER
I20260928 15:49:38.371510 2097 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd5754477d4df28044313a93f60af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260928 15:49:38.371771 2087 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:38.375358 2098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2fd5754477d4df28044313a93f60af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd5754477d4df28044313a93f60af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } } }
I20260928 15:49:38.375910 2098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:38.375324 2099 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2fd5754477d4df28044313a93f60af4. Latest consensus state: current_term: 1 leader_uuid: "d2fd5754477d4df28044313a93f60af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fd5754477d4df28044313a93f60af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } } }
I20260928 15:49:38.376677 2099 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:38.387570 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:38.364788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:52527 (local address 127.0.0.1:46307)
0928 15:49:38.365031 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:38.365037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:38.365417 (+ 380us) server_negotiation.cc:407] Connection header received
0928 15:49:38.365630 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:38.365637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:38.365740 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:38.365873 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:38.367673 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:38.368650 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:38.370946 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:38.371914 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:38.372033 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:38.376063 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:38.376076 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:38.376084 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:38.376138 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:38.381792 (+ 5654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:38.383562 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:38.383573 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:38.383580 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:38.383811 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:38.384790 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:38.384800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:38.384806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:38.385444 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:38.385858 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:38.386926 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0928 15:49:38.387209 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:49:38.387673 2104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:38.395104 2104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:38.404616 2104 catalog_manager.cc:1347] Generated new cluster ID: 9a5b26cc5d7e4e3e99d1357cd8839e4f
I20260928 15:49:38.404762 2104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:38.422245 2104 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:38.423609 2104 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:38.438573 2104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2fd5754477d4df28044313a93f60af4: Generated new TSK 0
I20260928 15:49:38.439455 2104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:38.447647 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2019
I20260928 15:49:38.449522 2082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:38.544876 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2019
W20260928 15:49:38.563462 2004 connection.cc:535] client connection to 127.0.0.1:46307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46307): Connection reset by peer (error 104)
2026-09-28T15:49:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2099 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:38 dist-test-slave-j968 krb5kdc[2119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:38 dist-test-slave-j968 krb5kdc[2119](info): set up 2 sockets
Sep 28 15:49:38 dist-test-slave-j968 krb5kdc[2119](info): commencing operation
krb5kdc: starting...
W20260928 15:49:39.666755 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:39 dist-test-slave-j968 krb5kdc[2119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:39.939177 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:40.295620 2132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:40.295990 2132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:40.296181 2132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:40.303555 2132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:40.303673 2132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:40.303764 2132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:40.303850 2132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:40.303957 2132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:40.314078 2132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:40.315505 2132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:40.316900 2132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:40.329962 2140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:40.329937 2138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:40.330003 2137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:40.331199 2132 server_base.cc:1034] running on GCE node
I20260928 15:49:40.332136 2132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:40.333804 2132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:40.335047 2132 hybrid_clock.cc:648] HybridClock initialized: now 1790610580334995 us; error 47 us; skew 500 ppm
I20260928 15:49:40.335421 2132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:40 dist-test-slave-j968 krb5kdc[2119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:40.338840 2132 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:40.340693 2132 webserver.cc:460] Webserver started at http://127.0.0.1:39397/ using document root <none> and password file <none>
I20260928 15:49:40.341434 2132 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:40.341585 2132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:40.341912 2132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:40.345103 2132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "4fefad2f4a5e4058861f5e61784f0242"
format_stamp: "Formatted at 2026-09-28 15:49:40 on dist-test-slave-j968"
server_key: "9ee1e3229af01fc492d58e388f693a18"
server_key_iv: "c8afdff9a16244564fa4f3817592da6c"
server_key_version: "encryptionkey@0"
I20260928 15:49:40.345922 2132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "4fefad2f4a5e4058861f5e61784f0242"
format_stamp: "Formatted at 2026-09-28 15:49:40 on dist-test-slave-j968"
server_key: "9ee1e3229af01fc492d58e388f693a18"
server_key_iv: "c8afdff9a16244564fa4f3817592da6c"
server_key_version: "encryptionkey@0"
I20260928 15:49:40.351367 2132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:49:40.355020 2147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:40.356689 2132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:40.356873 2132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "4fefad2f4a5e4058861f5e61784f0242"
format_stamp: "Formatted at 2026-09-28 15:49:40 on dist-test-slave-j968"
server_key: "9ee1e3229af01fc492d58e388f693a18"
server_key_iv: "c8afdff9a16244564fa4f3817592da6c"
server_key_version: "encryptionkey@0"
I20260928 15:49:40.357074 2132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:40.373481 2132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:40.378238 2132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:40.378595 2132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:40.399307 2132 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43671
I20260928 15:49:40.399338 2198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43671 every 8 connection(s)
I20260928 15:49:40.400810 2132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:40.404779 2199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:40.407222 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2132
I20260928 15:49:40.407437 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:40.407932 13333 external_mini_cluster.cc:1428] Setting key b4cbc908b0da35eeb8ffa412a5431032
I20260928 15:49:40.417028 2199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242: Bootstrap starting.
I20260928 15:49:40.420814 2199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:40.421844 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:40.409763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:44655 (local address 127.0.0.1:43671)
0928 15:49:40.410657 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:40.410668 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:40.410708 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:49:40.412452 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:40.412473 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:40.412891 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:40.413231 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:40.413834 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:40.415180 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:40.417118 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:40.417864 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:40.418006 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:40.419368 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:40.419375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:40.419389 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:40.419990 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:40.420075 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:40.420100 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:40.420219 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:40.420525 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 15:49:40.420718 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":419,"threads_started":1}
I20260928 15:49:40.421928 2199 log.cc:824] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:40.425009 2199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242: No bootstrap required, opened a new log
I20260928 15:49:40.426120 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:40.435451 2199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fefad2f4a5e4058861f5e61784f0242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260928 15:49:40.435954 2199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:40.436070 2199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fefad2f4a5e4058861f5e61784f0242, State: Initialized, Role: FOLLOWER
I20260928 15:49:40.436782 2199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fefad2f4a5e4058861f5e61784f0242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260928 15:49:40.437012 2199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:40.437180 2199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:40.437297 2199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:40.438582 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:40.431453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:47255 (local address 127.0.0.1:43671)
0928 15:49:40.431659 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:40.431665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:40.432250 (+ 585us) server_negotiation.cc:407] Connection header received
0928 15:49:40.432395 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:40.432400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:40.432497 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:40.432615 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:40.433116 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:40.434344 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:40.435767 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:40.436454 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:40.436574 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:40.437790 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:40.437796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:40.437800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:40.437822 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:40.437891 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:40.437917 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:40.438058 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:40.438189 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 15:49:40.438247 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:49:40.440389 2199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fefad2f4a5e4058861f5e61784f0242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260928 15:49:40.440843 2199 leader_election.cc:302] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fefad2f4a5e4058861f5e61784f0242; no voters:
I20260928 15:49:40.441442 2199 leader_election.cc:288] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:40.441640 2210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:40.442324 2210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [term 1 LEADER]: Becoming Leader. State: Replica: 4fefad2f4a5e4058861f5e61784f0242, State: Running, Role: LEADER
I20260928 15:49:40.443171 2210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fefad2f4a5e4058861f5e61784f0242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260928 15:49:40.444464 2199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:40.446090 2211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fefad2f4a5e4058861f5e61784f0242. Latest consensus state: current_term: 1 leader_uuid: "4fefad2f4a5e4058861f5e61784f0242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fefad2f4a5e4058861f5e61784f0242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } } }
I20260928 15:49:40.446117 2212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fefad2f4a5e4058861f5e61784f0242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fefad2f4a5e4058861f5e61784f0242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } } }
I20260928 15:49:40.446568 2211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:40.446576 2212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:40.458276 2215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:40.463508 2215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:40.472364 2215 catalog_manager.cc:1347] Generated new cluster ID: 685ff85f2fa947a584ab323cd9f2dc72
I20260928 15:49:40.472502 2215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:40.512394 2215 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:40.513814 2215 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:40.519558 2215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fefad2f4a5e4058861f5e61784f0242: Generated new TSK 0
I20260928 15:49:40.520483 2215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:40.528856 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2132
W20260928 15:49:40.530351 2194 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:44655
W20260928 15:49:40.531167 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2132: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:40.531322 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2132
W20260928 15:49:40.549685 2117 connection.cc:535] client connection to 127.0.0.1:43671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43671): Connection reset by peer (error 104)
2026-09-28T15:49:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:40 dist-test-slave-j968 krb5kdc[2231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:40 dist-test-slave-j968 krb5kdc[2231](info): set up 2 sockets
Sep 28 15:49:40 dist-test-slave-j968 krb5kdc[2231](info): commencing operation
krb5kdc: starting...
W20260928 15:49:41.654811 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:41 dist-test-slave-j968 krb5kdc[2231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:41.931375 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:42.279561 2244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:42.279891 2244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:42.279987 2244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:42.287436 2244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:42.287609 2244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:42.287676 2244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:42.287738 2244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:42.287782 2244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:42.298242 2244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:42.299610 2244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:42.300935 2244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:42.313223 2249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:42.313201 2250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:42.313279 2252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:42.314229 2244 server_base.cc:1034] running on GCE node
I20260928 15:49:42.315080 2244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:42.316713 2244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:42.317956 2244 hybrid_clock.cc:648] HybridClock initialized: now 1790610582317900 us; error 56 us; skew 500 ppm
I20260928 15:49:42.318367 2244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:42 dist-test-slave-j968 krb5kdc[2231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:42.329501 2244 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:42.331297 2244 webserver.cc:460] Webserver started at http://127.0.0.1:44689/ using document root <none> and password file <none>
I20260928 15:49:42.332178 2244 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:42.332338 2244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:42.332674 2244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:42.335842 2244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "1ee85d186bf04bae98a40020b3dd0d98"
format_stamp: "Formatted at 2026-09-28 15:49:42 on dist-test-slave-j968"
server_key: "56e92ded8337eef2770f2fcd5bec3575"
server_key_iv: "b4291d38d8cbc013ed32dfed9574a445"
server_key_version: "encryptionkey@0"
I20260928 15:49:42.336805 2244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "1ee85d186bf04bae98a40020b3dd0d98"
format_stamp: "Formatted at 2026-09-28 15:49:42 on dist-test-slave-j968"
server_key: "56e92ded8337eef2770f2fcd5bec3575"
server_key_iv: "b4291d38d8cbc013ed32dfed9574a445"
server_key_version: "encryptionkey@0"
I20260928 15:49:42.342541 2244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:49:42.346560 2259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:42.348207 2244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:49:42.348408 2244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "1ee85d186bf04bae98a40020b3dd0d98"
format_stamp: "Formatted at 2026-09-28 15:49:42 on dist-test-slave-j968"
server_key: "56e92ded8337eef2770f2fcd5bec3575"
server_key_iv: "b4291d38d8cbc013ed32dfed9574a445"
server_key_version: "encryptionkey@0"
I20260928 15:49:42.348650 2244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:42.354609 2244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:42.358855 2244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:42.359128 2244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:42.378821 2244 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39271
I20260928 15:49:42.378859 2310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39271 every 8 connection(s)
I20260928 15:49:42.380394 2244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:42.384452 2311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:42.389678 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2244
I20260928 15:49:42.389851 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:42.390305 13333 external_mini_cluster.cc:1428] Setting key 7cc307c7a91dc4d85d2505e771c61f5f
I20260928 15:49:42.397034 2311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98: Bootstrap starting.
I20260928 15:49:42.399657 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:42.392285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:60977 (local address 127.0.0.1:39271)
0928 15:49:42.393158 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:42.393180 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:42.393235 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:49:42.395689 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:42.395718 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:42.396195 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:42.396635 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:42.397655 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:42.397665 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:42.397683 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:42.397701 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:42.397771 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:42.397796 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:42.398019 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:42.398146 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 15:49:42.398334 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":369,"threads_started":1}
I20260928 15:49:42.400985 2311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:42.402175 2311 log.cc:824] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:42.403640 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:42.405598 2311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98: No bootstrap required, opened a new log
I20260928 15:49:42.411419 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:42.408864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:54989 (local address 127.0.0.1:39271)
0928 15:49:42.409089 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:42.409097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:42.409489 (+ 392us) server_negotiation.cc:407] Connection header received
0928 15:49:42.409660 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:42.409664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:42.409714 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:42.409818 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:42.410736 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:42.410740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:42.410743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:42.410758 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:42.410795 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:42.410809 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:42.410896 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:42.411030 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 15:49:42.411079 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:49:42.415591 2311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee85d186bf04bae98a40020b3dd0d98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260928 15:49:42.416011 2311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:42.416133 2311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee85d186bf04bae98a40020b3dd0d98, State: Initialized, Role: FOLLOWER
I20260928 15:49:42.416810 2311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee85d186bf04bae98a40020b3dd0d98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260928 15:49:42.417207 2311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:42.417399 2311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:42.417600 2311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:42.419921 2311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee85d186bf04bae98a40020b3dd0d98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260928 15:49:42.420450 2311 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee85d186bf04bae98a40020b3dd0d98; no voters:
I20260928 15:49:42.421096 2311 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:42.421235 2321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:42.421968 2321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [term 1 LEADER]: Becoming Leader. State: Replica: 1ee85d186bf04bae98a40020b3dd0d98, State: Running, Role: LEADER
I20260928 15:49:42.422513 2321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee85d186bf04bae98a40020b3dd0d98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } }
I20260928 15:49:42.423336 2311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:42.425838 2323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee85d186bf04bae98a40020b3dd0d98. Latest consensus state: current_term: 1 leader_uuid: "1ee85d186bf04bae98a40020b3dd0d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee85d186bf04bae98a40020b3dd0d98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } } }
I20260928 15:49:42.426302 2323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:42.426211 2322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee85d186bf04bae98a40020b3dd0d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee85d186bf04bae98a40020b3dd0d98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39271 } } }
I20260928 15:49:42.426482 2322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:42.428056 2330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:42.433480 2330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:42.443851 2330 catalog_manager.cc:1347] Generated new cluster ID: 763c0a00db044faa88c4caf3bc9e1575
I20260928 15:49:42.444015 2330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:42.470408 2330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:42.470875 2330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:42.484707 2330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ee85d186bf04bae98a40020b3dd0d98: Generated new TSK 0
I20260928 15:49:42.485576 2330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:42.498210 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2244
W20260928 15:49:42.499645 2306 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:60977
W20260928 15:49:42.500429 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2244: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:42.500604 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2244
2026-09-28T15:49:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:42 dist-test-slave-j968 krb5kdc[2342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:42 dist-test-slave-j968 krb5kdc[2342](info): set up 2 sockets
Sep 28 15:49:42 dist-test-slave-j968 krb5kdc[2342](info): commencing operation
krb5kdc: starting...
W20260928 15:49:43.618852 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:43 dist-test-slave-j968 krb5kdc[2342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:43.897027 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:44.246256 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:44.246562 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:44.246742 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:44.254016 2355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:44.254123 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:44.254242 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:44.254347 2355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:44.254429 2355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:44.264499 2355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:44.265832 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:44.267174 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:44.279539 2364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:44.279623 2361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:44.279597 2362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:44.280496 2355 server_base.cc:1034] running on GCE node
I20260928 15:49:44.281486 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:44.283360 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:44.284617 2355 hybrid_clock.cc:648] HybridClock initialized: now 1790610584284555 us; error 35 us; skew 500 ppm
I20260928 15:49:44.284947 2355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:44 dist-test-slave-j968 krb5kdc[2342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:44.288357 2355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:44.290531 2355 webserver.cc:460] Webserver started at http://127.0.0.1:37729/ using document root <none> and password file <none>
I20260928 15:49:44.291328 2355 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:44.291484 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:44.291811 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:44.295042 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "cb620667ad804be2b2e147652393abc6"
format_stamp: "Formatted at 2026-09-28 15:49:44 on dist-test-slave-j968"
server_key: "dc56739e85e84419fb821783b4413f23"
server_key_iv: "66fe8abb4a6f63e35d011c8382a553b4"
server_key_version: "encryptionkey@0"
I20260928 15:49:44.295838 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "cb620667ad804be2b2e147652393abc6"
format_stamp: "Formatted at 2026-09-28 15:49:44 on dist-test-slave-j968"
server_key: "dc56739e85e84419fb821783b4413f23"
server_key_iv: "66fe8abb4a6f63e35d011c8382a553b4"
server_key_version: "encryptionkey@0"
I20260928 15:49:44.301548 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:49:44.305249 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:44.306694 2355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:44.306885 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "cb620667ad804be2b2e147652393abc6"
format_stamp: "Formatted at 2026-09-28 15:49:44 on dist-test-slave-j968"
server_key: "dc56739e85e84419fb821783b4413f23"
server_key_iv: "66fe8abb4a6f63e35d011c8382a553b4"
server_key_version: "encryptionkey@0"
I20260928 15:49:44.307102 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:44.328925 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:44.333629 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:44.333918 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:44.353466 2355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40097
I20260928 15:49:44.353562 2422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40097 every 8 connection(s)
I20260928 15:49:44.354883 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:44.358723 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:44.364190 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2355
I20260928 15:49:44.364384 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:44.364840 13333 external_mini_cluster.cc:1428] Setting key f67c59b4afc26e33d1a83da99e6b1509
I20260928 15:49:44.369657 2423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6: Bootstrap starting.
I20260928 15:49:44.373485 2423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:44.374739 2423 log.cc:824] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:44.378398 2423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6: No bootstrap required, opened a new log
Sep 28 15:49:44 dist-test-slave-j968 krb5kdc[2342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:49:44.386852 2423 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb620667ad804be2b2e147652393abc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260928 15:49:44.387284 2423 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:44.387413 2423 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb620667ad804be2b2e147652393abc6, State: Initialized, Role: FOLLOWER
I20260928 15:49:44.388136 2423 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb620667ad804be2b2e147652393abc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260928 15:49:44.388455 2423 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:44.388618 2423 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:44.388804 2423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:44.389573 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:44.366687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:36315 (local address 127.0.0.1:40097)
0928 15:49:44.367570 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:44.367581 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:44.367628 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:49:44.369285 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:44.369310 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:44.369696 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:44.370062 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:44.371992 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:44.373256 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:44.375080 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:44.376230 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:44.376359 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:44.381400 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:44.381408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:44.381424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:44.381471 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:44.384892 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:44.385820 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:44.385828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:44.385834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:44.386022 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:44.386493 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:44.386498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:44.386501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:44.387057 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:44.387427 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:44.388026 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 15:49:44.388347 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":405,"threads_started":1}
I20260928 15:49:44.391338 2423 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb620667ad804be2b2e147652393abc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260928 15:49:44.391865 2423 leader_election.cc:302] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb620667ad804be2b2e147652393abc6; no voters:
I20260928 15:49:44.392558 2423 leader_election.cc:288] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:44.392722 2428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:44.393951 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:44.393889 2428 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [term 1 LEADER]: Becoming Leader. State: Replica: cb620667ad804be2b2e147652393abc6, State: Running, Role: LEADER
I20260928 15:49:44.394448 2428 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb620667ad804be2b2e147652393abc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260928 15:49:44.394918 2423 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:44.397282 2429 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb620667ad804be2b2e147652393abc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb620667ad804be2b2e147652393abc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } } }
I20260928 15:49:44.397773 2429 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:44.399405 2430 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb620667ad804be2b2e147652393abc6. Latest consensus state: current_term: 1 leader_uuid: "cb620667ad804be2b2e147652393abc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb620667ad804be2b2e147652393abc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } } }
I20260928 15:49:44.406107 2430 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:44.413182 2436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:44.419418 2436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:44.434748 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:44.411968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:50091 (local address 127.0.0.1:40097)
0928 15:49:44.412952 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:44.412972 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:44.415829 (+ 2857us) server_negotiation.cc:407] Connection header received
0928 15:49:44.416913 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:44.416919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:44.417011 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:44.417086 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:44.419413 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:44.420643 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:44.422480 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:44.423160 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:44.423270 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:44.426630 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:44.426640 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:44.426647 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:44.426692 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:44.430522 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:44.431544 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:44.431550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:44.431554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:44.431700 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:44.432305 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:44.432311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:44.432315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:44.432708 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:44.432992 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:44.434204 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0928 15:49:44.434388 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804}
I20260928 15:49:44.448320 2436 catalog_manager.cc:1347] Generated new cluster ID: 6bc10bf31b464dd3bf163c26d20c6f7e
I20260928 15:49:44.448757 2436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:44.464924 2436 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:44.466318 2436 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:44.472344 2436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb620667ad804be2b2e147652393abc6: Generated new TSK 0
I20260928 15:49:44.473542 2436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:44.485711 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2355
I20260928 15:49:44.487054 2418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:44.580792 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2355
W20260928 15:49:44.599128 2340 connection.cc:535] client connection to 127.0.0.1:40097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40097): Connection reset by peer (error 104)
2026-09-28T15:49:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:44 dist-test-slave-j968 krb5kdc[2455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:44 dist-test-slave-j968 krb5kdc[2455](info): set up 2 sockets
Sep 28 15:49:44 dist-test-slave-j968 krb5kdc[2455](info): commencing operation
krb5kdc: starting...
W20260928 15:49:45.714854 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:45 dist-test-slave-j968 krb5kdc[2455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:45.983527 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:46.330583 2468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:46.330966 2468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:46.331164 2468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:46.339004 2468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:46.339192 2468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:46.339299 2468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:46.339409 2468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:46.339488 2468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:46.349860 2468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:46.351253 2468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:46.352661 2468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:46.364651 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:46.364724 2476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:46.364676 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:46.365890 2468 server_base.cc:1034] running on GCE node
I20260928 15:49:46.366750 2468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:46.368250 2468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:46.369478 2468 hybrid_clock.cc:648] HybridClock initialized: now 1790610586369443 us; error 35 us; skew 500 ppm
I20260928 15:49:46.369836 2468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:46 dist-test-slave-j968 krb5kdc[2455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:46.373358 2468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:46.375077 2468 webserver.cc:460] Webserver started at http://127.0.0.1:45281/ using document root <none> and password file <none>
I20260928 15:49:46.375849 2468 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:46.376070 2468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:46.376415 2468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:46.379510 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "5c7aaf33b2a745d7b8856aa5be4e406d"
format_stamp: "Formatted at 2026-09-28 15:49:46 on dist-test-slave-j968"
server_key: "78e05ae9e07c05bac0dd38e34b6416a1"
server_key_iv: "aa2f5623a8379da386cfdcc53a5d5610"
server_key_version: "encryptionkey@0"
I20260928 15:49:46.380324 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "5c7aaf33b2a745d7b8856aa5be4e406d"
format_stamp: "Formatted at 2026-09-28 15:49:46 on dist-test-slave-j968"
server_key: "78e05ae9e07c05bac0dd38e34b6416a1"
server_key_iv: "aa2f5623a8379da386cfdcc53a5d5610"
server_key_version: "encryptionkey@0"
I20260928 15:49:46.385742 2468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:49:46.389360 2483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:46.390820 2468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:46.391001 2468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "5c7aaf33b2a745d7b8856aa5be4e406d"
format_stamp: "Formatted at 2026-09-28 15:49:46 on dist-test-slave-j968"
server_key: "78e05ae9e07c05bac0dd38e34b6416a1"
server_key_iv: "aa2f5623a8379da386cfdcc53a5d5610"
server_key_version: "encryptionkey@0"
I20260928 15:49:46.391211 2468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:46.409651 2468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:46.414294 2468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:46.414601 2468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:46.433840 2468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38635
I20260928 15:49:46.433864 2535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38635 every 8 connection(s)
I20260928 15:49:46.435348 2468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:46.439252 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:46.439927 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2468
I20260928 15:49:46.440137 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:46.440587 13333 external_mini_cluster.cc:1428] Setting key 52ca70c3ca562f90eaf712c9614e3c8b
I20260928 15:49:46.453207 2536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d: Bootstrap starting.
I20260928 15:49:46.454627 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:46.442503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:59673 (local address 127.0.0.1:38635)
0928 15:49:46.443332 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:46.443344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:46.443385 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:49:46.445375 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:46.445401 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:46.445820 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:46.446195 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:46.446779 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:46.448102 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:46.450060 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:46.450758 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:46.450890 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:46.452196 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:46.452204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:46.452219 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:46.452793 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:46.452879 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:46.452904 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:46.453024 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:46.453342 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 15:49:46.453566 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":369,"threads_started":1}
I20260928 15:49:46.456810 2536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:46.458153 2536 log.cc:824] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:46.458858 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:46.461175 2536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d: No bootstrap required, opened a new log
I20260928 15:49:46.470263 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:46.463541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:56321 (local address 127.0.0.1:38635)
0928 15:49:46.463752 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:46.463763 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:46.464113 (+ 350us) server_negotiation.cc:407] Connection header received
0928 15:49:46.464282 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:46.464293 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:46.464380 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:46.464495 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:46.464945 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:46.466112 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:46.467442 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:46.468235 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:46.468373 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:46.469459 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:46.469466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:46.469473 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:46.469499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:46.469558 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:46.469585 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:46.469707 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:46.469868 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 15:49:46.469933 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:49:46.471096 2536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7aaf33b2a745d7b8856aa5be4e406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } }
I20260928 15:49:46.471518 2536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:46.471647 2536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7aaf33b2a745d7b8856aa5be4e406d, State: Initialized, Role: FOLLOWER
I20260928 15:49:46.472359 2536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7aaf33b2a745d7b8856aa5be4e406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } }
I20260928 15:49:46.472615 2536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:46.472786 2536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:46.472954 2536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:46.475232 2536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7aaf33b2a745d7b8856aa5be4e406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } }
I20260928 15:49:46.475786 2536 leader_election.cc:302] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c7aaf33b2a745d7b8856aa5be4e406d; no voters:
I20260928 15:49:46.476450 2536 leader_election.cc:288] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:46.476611 2546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:46.477813 2546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [term 1 LEADER]: Becoming Leader. State: Replica: 5c7aaf33b2a745d7b8856aa5be4e406d, State: Running, Role: LEADER
I20260928 15:49:46.478497 2546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7aaf33b2a745d7b8856aa5be4e406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } }
I20260928 15:49:46.478866 2536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:46.481966 2548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c7aaf33b2a745d7b8856aa5be4e406d. Latest consensus state: current_term: 1 leader_uuid: "5c7aaf33b2a745d7b8856aa5be4e406d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7aaf33b2a745d7b8856aa5be4e406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } } }
I20260928 15:49:46.482378 2548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:46.481732 2547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c7aaf33b2a745d7b8856aa5be4e406d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7aaf33b2a745d7b8856aa5be4e406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } } }
I20260928 15:49:46.482550 2547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:46.483929 2555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:46.489735 2555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:46.500022 2555 catalog_manager.cc:1347] Generated new cluster ID: 965d7a3bb8fd41bd839f84b3daaba83c
I20260928 15:49:46.500160 2555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:46.519309 2555 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:46.520629 2555 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:46.531117 2555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c7aaf33b2a745d7b8856aa5be4e406d: Generated new TSK 0
I20260928 15:49:46.532011 2555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:46.538506 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2468
W20260928 15:49:46.539705 2531 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:59673
W20260928 15:49:46.540614 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2468: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:46.540778 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2468
W20260928 15:49:46.561009 2453 connection.cc:535] client connection to 127.0.0.1:38635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38635): Connection reset by peer (error 104)
2026-09-28T15:49:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:46 dist-test-slave-j968 krb5kdc[2567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:46 dist-test-slave-j968 krb5kdc[2567](info): set up 2 sockets
Sep 28 15:49:46 dist-test-slave-j968 krb5kdc[2567](info): commencing operation
krb5kdc: starting...
W20260928 15:49:47.658793 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:47 dist-test-slave-j968 krb5kdc[2567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:49:47.933717 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:48.279578 2580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:48.279983 2580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:48.280149 2580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:48.287506 2580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:48.287667 2580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:48.287786 2580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:48.287914 2580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:48.287984 2580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:48.298590 2580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:48.300175 2580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:48.301602 2580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:48.314020 2585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:48.314091 2586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:48.314056 2588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:48.315346 2580 server_base.cc:1034] running on GCE node
I20260928 15:49:48.316293 2580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:48.317792 2580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:48.319042 2580 hybrid_clock.cc:648] HybridClock initialized: now 1790610588318997 us; error 39 us; skew 500 ppm
I20260928 15:49:48.319443 2580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:48 dist-test-slave-j968 krb5kdc[2567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:48.327332 2580 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:49:48.329141 2580 webserver.cc:460] Webserver started at http://127.0.0.1:41961/ using document root <none> and password file <none>
I20260928 15:49:48.329833 2580 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:48.329953 2580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:48.330229 2580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:48.333493 2580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "11e4be1bbcc14779a644c71567f77ca9"
format_stamp: "Formatted at 2026-09-28 15:49:48 on dist-test-slave-j968"
server_key: "4b2307cd5d14cd88750db016dce54708"
server_key_iv: "261adb766c96b6cc95608610bacc2599"
server_key_version: "encryptionkey@0"
I20260928 15:49:48.334256 2580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "11e4be1bbcc14779a644c71567f77ca9"
format_stamp: "Formatted at 2026-09-28 15:49:48 on dist-test-slave-j968"
server_key: "4b2307cd5d14cd88750db016dce54708"
server_key_iv: "261adb766c96b6cc95608610bacc2599"
server_key_version: "encryptionkey@0"
I20260928 15:49:48.340054 2580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:49:48.343910 2595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:48.345743 2580 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:49:48.345973 2580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "11e4be1bbcc14779a644c71567f77ca9"
format_stamp: "Formatted at 2026-09-28 15:49:48 on dist-test-slave-j968"
server_key: "4b2307cd5d14cd88750db016dce54708"
server_key_iv: "261adb766c96b6cc95608610bacc2599"
server_key_version: "encryptionkey@0"
I20260928 15:49:48.346215 2580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:48.352962 2580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:48.357870 2580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:48.358217 2580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:48.379055 2580 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34775
I20260928 15:49:48.379040 2646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34775 every 8 connection(s)
I20260928 15:49:48.380925 2580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:48.385214 2647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:48.390288 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2580
I20260928 15:49:48.390506 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:48.391115 13333 external_mini_cluster.cc:1428] Setting key 61092de7773ee7a25f279a3cf6cf6d22
I20260928 15:49:48.397704 2647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9: Bootstrap starting.
I20260928 15:49:48.399752 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:48.393236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:36509 (local address 127.0.0.1:34775)
0928 15:49:48.394143 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:48.394155 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:48.394200 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:49:48.396141 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:48.396167 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:48.396539 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:48.396878 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:48.398035 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:48.398042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:48.398056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:48.398071 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:48.398135 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:48.398154 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:48.398334 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:48.398460 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 15:49:48.398711 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":406,"threads_started":1}
I20260928 15:49:48.401672 2647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:48.402972 2647 log.cc:824] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:48.403977 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:48.406117 2647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9: No bootstrap required, opened a new log
I20260928 15:49:48.412361 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:48.409549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:42715 (local address 127.0.0.1:34775)
0928 15:49:48.409799 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:48.409810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:48.410056 (+ 246us) server_negotiation.cc:407] Connection header received
0928 15:49:48.410304 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:48.410309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:48.410382 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:48.410501 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:48.411666 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:48.411670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:48.411674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:48.411690 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:48.411733 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:48.411748 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:48.411916 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:48.412009 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 15:49:48.412084 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:49:48.416035 2647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e4be1bbcc14779a644c71567f77ca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34775 } }
I20260928 15:49:48.416492 2647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:48.416610 2647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e4be1bbcc14779a644c71567f77ca9, State: Initialized, Role: FOLLOWER
I20260928 15:49:48.417153 2647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e4be1bbcc14779a644c71567f77ca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34775 } }
I20260928 15:49:48.417361 2647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:48.417461 2647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:48.417596 2647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:48.419827 2647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e4be1bbcc14779a644c71567f77ca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34775 } }
I20260928 15:49:48.420394 2647 leader_election.cc:302] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11e4be1bbcc14779a644c71567f77ca9; no voters:
I20260928 15:49:48.421057 2647 leader_election.cc:288] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:48.421204 2657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:48.422250 2657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [term 1 LEADER]: Becoming Leader. State: Replica: 11e4be1bbcc14779a644c71567f77ca9, State: Running, Role: LEADER
I20260928 15:49:48.422844 2657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e4be1bbcc14779a644c71567f77ca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34775 } }
I20260928 15:49:48.423446 2647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:48.425874 2658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11e4be1bbcc14779a644c71567f77ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e4be1bbcc14779a644c71567f77ca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34775 } } }
I20260928 15:49:48.425860 2659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11e4be1bbcc14779a644c71567f77ca9. Latest consensus state: current_term: 1 leader_uuid: "11e4be1bbcc14779a644c71567f77ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e4be1bbcc14779a644c71567f77ca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34775 } } }
I20260928 15:49:48.426388 2659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:48.426388 2658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:48.431034 2664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:48.436308 2664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:48.445042 2664 catalog_manager.cc:1347] Generated new cluster ID: 8752b7bd389748009d54fc10d8652e33
I20260928 15:49:48.445179 2664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:48.467201 2664 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:48.467710 2664 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:48.475245 2664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11e4be1bbcc14779a644c71567f77ca9: Generated new TSK 0
I20260928 15:49:48.476138 2664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:48.482689 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2580
W20260928 15:49:48.484294 2642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.152:36509
W20260928 15:49:48.485005 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2580: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:48.485163 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2580
2026-09-28T15:49:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:48 dist-test-slave-j968 krb5kdc[2678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:48 dist-test-slave-j968 krb5kdc[2678](info): set up 2 sockets
Sep 28 15:49:48 dist-test-slave-j968 krb5kdc[2678](info): commencing operation
krb5kdc: starting...
W20260928 15:49:49.618764 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:49 dist-test-slave-j968 krb5kdc[2678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:49.895627 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:35927
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:35927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:50.244465 2691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:50.244768 2691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:50.244930 2691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:50.253408 2691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:50.253523 2691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:50.253633 2691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:50.253737 2691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:50.253800 2691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:50.264146 2691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:35927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:35927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:50.265571 2691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:50.266922 2691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:50.279191 2696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:50.279260 2699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:50.279284 2697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:50.279925 2691 server_base.cc:1034] running on GCE node
I20260928 15:49:50.280802 2691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:50.282228 2691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:50.283439 2691 hybrid_clock.cc:648] HybridClock initialized: now 1790610590283400 us; error 33 us; skew 500 ppm
I20260928 15:49:50.283814 2691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:50 dist-test-slave-j968 krb5kdc[2678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610590, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:50.287094 2691 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:49:50.288620 2691 webserver.cc:460] Webserver started at http://127.13.5.126:40681/ using document root <none> and password file <none>
I20260928 15:49:50.289261 2691 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:50.289361 2691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:50.289621 2691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:50.292687 2691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "83955a90e39c46fdb50de3fde72050c3"
format_stamp: "Formatted at 2026-09-28 15:49:50 on dist-test-slave-j968"
server_key: "4844434dad7a521b6bd609e943cddaac"
server_key_iv: "a7dcc6178e35f86d32ba4ab9ef781b6b"
server_key_version: "encryptionkey@0"
I20260928 15:49:50.293435 2691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "83955a90e39c46fdb50de3fde72050c3"
format_stamp: "Formatted at 2026-09-28 15:49:50 on dist-test-slave-j968"
server_key: "4844434dad7a521b6bd609e943cddaac"
server_key_iv: "a7dcc6178e35f86d32ba4ab9ef781b6b"
server_key_version: "encryptionkey@0"
I20260928 15:49:50.298888 2691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 15:49:50.302510 2706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:50.304087 2691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:49:50.304291 2691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "83955a90e39c46fdb50de3fde72050c3"
format_stamp: "Formatted at 2026-09-28 15:49:50 on dist-test-slave-j968"
server_key: "4844434dad7a521b6bd609e943cddaac"
server_key_iv: "a7dcc6178e35f86d32ba4ab9ef781b6b"
server_key_version: "encryptionkey@0"
I20260928 15:49:50.304512 2691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:50.337631 2691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:50.342558 2691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:50.342900 2691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:50.362368 2691 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:35927
I20260928 15:49:50.362398 2757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:35927 every 8 connection(s)
I20260928 15:49:50.363919 2691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:50.367794 2758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:50.372289 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2691
I20260928 15:49:50.372488 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:50.373055 13333 external_mini_cluster.cc:1428] Setting key 626e69678750783141fc23c369e7f086
I20260928 15:49:50.379956 2758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3: Bootstrap starting.
I20260928 15:49:50.383924 2758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:50.385298 2758 log.cc:824] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:50.389021 2758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3: No bootstrap required, opened a new log
Sep 28 15:49:50 dist-test-slave-j968 krb5kdc[2678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:50.400166 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:50.374959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49911 (local address 127.13.5.126:35927)
0928 15:49:50.375736 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:50.375748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:50.375795 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:49:50.378248 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:50.378274 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:50.378699 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:50.379036 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:50.380895 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:50.382355 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:50.384214 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:50.385115 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:50.385264 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:50.390364 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:50.390375 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:50.390394 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:50.390448 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:50.394827 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:50.395818 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:50.395827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:50.395836 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:50.396092 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:50.396575 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:50.396580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:50.396584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:50.397372 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:50.397709 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:50.398196 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 15:49:50.398596 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":366,"threads_started":1}
I20260928 15:49:50.400672 2758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83955a90e39c46fdb50de3fde72050c3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35927 } }
I20260928 15:49:50.401093 2758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:50.401216 2758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83955a90e39c46fdb50de3fde72050c3, State: Initialized, Role: FOLLOWER
I20260928 15:49:50.401882 2758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83955a90e39c46fdb50de3fde72050c3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35927 } }
I20260928 15:49:50.402166 2758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:50.402323 2758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:50.402509 2758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:50.404592 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:50.405450 2758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83955a90e39c46fdb50de3fde72050c3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35927 } }
I20260928 15:49:50.406051 2758 leader_election.cc:302] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83955a90e39c46fdb50de3fde72050c3; no voters:
I20260928 15:49:50.406730 2758 leader_election.cc:288] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:50.406860 2763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:50.408213 2763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [term 1 LEADER]: Becoming Leader. State: Replica: 83955a90e39c46fdb50de3fde72050c3, State: Running, Role: LEADER
I20260928 15:49:50.408838 2763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83955a90e39c46fdb50de3fde72050c3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35927 } }
I20260928 15:49:50.409503 2758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:50.411691 2765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83955a90e39c46fdb50de3fde72050c3. Latest consensus state: current_term: 1 leader_uuid: "83955a90e39c46fdb50de3fde72050c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83955a90e39c46fdb50de3fde72050c3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35927 } } }
I20260928 15:49:50.412150 2765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:50.411720 2764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83955a90e39c46fdb50de3fde72050c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83955a90e39c46fdb50de3fde72050c3" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35927 } } }
I20260928 15:49:50.412783 2764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:50.413599 2771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:50.422631 2771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:50.432946 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:50.414186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52085 (local address 127.13.5.126:35927)
0928 15:49:50.414366 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:50.414378 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:50.415016 (+ 638us) server_negotiation.cc:407] Connection header received
0928 15:49:50.415210 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:50.415218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:50.415308 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:50.415418 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:50.417450 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:50.418714 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:50.420656 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:50.421360 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:50.421494 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:50.425024 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:50.425031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:50.425037 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:50.425079 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:50.428982 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:50.430366 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:50.430372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:50.430375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:50.430532 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:50.431078 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:50.431084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:50.431086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:50.431505 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:50.431775 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:50.432326 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 15:49:50.432528 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:49:50.446628 2771 catalog_manager.cc:1347] Generated new cluster ID: 81aa567f75fe4faf910e497c011606ed
I20260928 15:49:50.446759 2771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:50.468291 2771 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:50.470263 2771 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:50.480901 2771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83955a90e39c46fdb50de3fde72050c3: Generated new TSK 0
I20260928 15:49:50.482131 2771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:50.494375 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2691
I20260928 15:49:50.496117 2753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:50.593410 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2691
W20260928 15:49:50.612289 2676 connection.cc:535] client connection to 127.13.5.126:35927 recv error: Network error: failed to read from TLS socket (remote: 127.13.5.126:35927): Connection reset by peer (error 104)
2026-09-28T15:49:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2109 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:50 dist-test-slave-j968 krb5kdc[2791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:50 dist-test-slave-j968 krb5kdc[2791](info): set up 2 sockets
Sep 28 15:49:50 dist-test-slave-j968 krb5kdc[2791](info): commencing operation
krb5kdc: starting...
W20260928 15:49:51.718847 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:51 dist-test-slave-j968 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:51.992357 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:35731
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:52.339293 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:52.339607 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:52.339701 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:52.347288 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:52.347414 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:52.347462 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:52.347517 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:52.347553 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:52.357908 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:35731
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:52.359201 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:52.360574 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:52.373286 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:52.373390 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:52.373265 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:52.377789 2804 server_base.cc:1034] running on GCE node
I20260928 15:49:52.378690 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:52.380236 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:52.381479 2804 hybrid_clock.cc:648] HybridClock initialized: now 1790610592381424 us; error 37 us; skew 500 ppm
I20260928 15:49:52.381907 2804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:52 dist-test-slave-j968 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610592, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:52.385187 2804 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:49:52.386741 2804 webserver.cc:460] Webserver started at http://127.13.5.126:40197/ using document root <none> and password file <none>
I20260928 15:49:52.387516 2804 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:52.387666 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:52.388058 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:52.391324 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "43902dcffaea49748a6d24f77f5f83a6"
format_stamp: "Formatted at 2026-09-28 15:49:52 on dist-test-slave-j968"
server_key: "084378112f700ab6b76f4dd0b53cfe2c"
server_key_iv: "2a931635759a8cffac6f35f2f47be7af"
server_key_version: "encryptionkey@0"
I20260928 15:49:52.392190 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "43902dcffaea49748a6d24f77f5f83a6"
format_stamp: "Formatted at 2026-09-28 15:49:52 on dist-test-slave-j968"
server_key: "084378112f700ab6b76f4dd0b53cfe2c"
server_key_iv: "2a931635759a8cffac6f35f2f47be7af"
server_key_version: "encryptionkey@0"
I20260928 15:49:52.397468 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:49:52.401116 2819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:52.402550 2804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:52.402743 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "43902dcffaea49748a6d24f77f5f83a6"
format_stamp: "Formatted at 2026-09-28 15:49:52 on dist-test-slave-j968"
server_key: "084378112f700ab6b76f4dd0b53cfe2c"
server_key_iv: "2a931635759a8cffac6f35f2f47be7af"
server_key_version: "encryptionkey@0"
I20260928 15:49:52.402956 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:52.415465 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:52.419574 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:52.419974 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:52.439327 2804 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:35731
I20260928 15:49:52.439376 2871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:35731 every 8 connection(s)
I20260928 15:49:52.440861 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:52.444798 2872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:52.449383 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2804
I20260928 15:49:52.449580 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:52.450127 13333 external_mini_cluster.cc:1428] Setting key 2269523b055a209c9d4567fa9f16d406
I20260928 15:49:52.458568 2872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6: Bootstrap starting.
I20260928 15:49:52.462481 2872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:52.463660 2872 log.cc:824] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:52.464093 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:52.452053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58687 (local address 127.13.5.126:35731)
0928 15:49:52.452858 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:52.452870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:52.452911 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:49:52.454564 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:52.454587 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:52.455007 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:52.455336 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:52.455984 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:52.457364 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:52.459259 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:52.460028 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:52.460163 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:52.461604 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:52.461612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:52.461627 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:52.462165 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:52.462248 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:52.462272 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:52.462395 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:52.462779 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 15:49:52.462984 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":391,"threads_started":1}
I20260928 15:49:52.466682 2872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6: No bootstrap required, opened a new log
I20260928 15:49:52.468276 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:52.477317 2872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43902dcffaea49748a6d24f77f5f83a6" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35731 } }
I20260928 15:49:52.477736 2872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:52.477841 2872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43902dcffaea49748a6d24f77f5f83a6, State: Initialized, Role: FOLLOWER
I20260928 15:49:52.478596 2872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43902dcffaea49748a6d24f77f5f83a6" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35731 } }
I20260928 15:49:52.478886 2872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:52.479055 2872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:52.479231 2872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:52.480999 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:52.473861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46679 (local address 127.13.5.126:35731)
0928 15:49:52.474087 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:52.474093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:52.474551 (+ 458us) server_negotiation.cc:407] Connection header received
0928 15:49:52.474690 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:52.474701 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:52.474794 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:52.474893 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:52.475427 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:52.476653 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:52.478134 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:52.478775 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:52.478911 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:52.480208 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:52.480214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:52.480220 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:52.480241 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:52.480309 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:52.480329 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:52.480445 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:52.480599 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 15:49:52.480662 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:49:52.482164 2872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43902dcffaea49748a6d24f77f5f83a6" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35731 } }
I20260928 15:49:52.482697 2872 leader_election.cc:302] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43902dcffaea49748a6d24f77f5f83a6; no voters:
I20260928 15:49:52.483248 2872 leader_election.cc:288] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:52.483388 2882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:52.484155 2882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [term 1 LEADER]: Becoming Leader. State: Replica: 43902dcffaea49748a6d24f77f5f83a6, State: Running, Role: LEADER
I20260928 15:49:52.484737 2882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43902dcffaea49748a6d24f77f5f83a6" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35731 } }
I20260928 15:49:52.485489 2872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:52.487463 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43902dcffaea49748a6d24f77f5f83a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43902dcffaea49748a6d24f77f5f83a6" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35731 } } }
I20260928 15:49:52.487887 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:52.488060 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43902dcffaea49748a6d24f77f5f83a6. Latest consensus state: current_term: 1 leader_uuid: "43902dcffaea49748a6d24f77f5f83a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43902dcffaea49748a6d24f77f5f83a6" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 35731 } } }
I20260928 15:49:52.488396 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:52.493214 2891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:52.497766 2891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:52.506371 2891 catalog_manager.cc:1347] Generated new cluster ID: 2a776246a4054f2eb794efb53cd7157d
I20260928 15:49:52.506487 2891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:52.542371 2891 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:52.543848 2891 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:52.551800 2891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43902dcffaea49748a6d24f77f5f83a6: Generated new TSK 0
I20260928 15:49:52.552752 2891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:52.559998 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2804
W20260928 15:49:52.561210 2867 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58687
W20260928 15:49:52.562067 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2804: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:52.562232 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2804
W20260928 15:49:52.580633 2789 connection.cc:535] client connection to 127.13.5.126:35731 recv error: Network error: failed to read from TLS socket (remote: 127.13.5.126:35731): Connection reset by peer (error 104)
2026-09-28T15:49:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:52 dist-test-slave-j968 krb5kdc[2903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:52 dist-test-slave-j968 krb5kdc[2903](info): set up 2 sockets
Sep 28 15:49:52 dist-test-slave-j968 krb5kdc[2903](info): commencing operation
krb5kdc: starting...
W20260928 15:49:53.703751 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:53 dist-test-slave-j968 krb5kdc[2903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:53.980271 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36255
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:36255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:54.330955 2916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:54.331285 2916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:54.331473 2916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:54.339033 2916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:54.339151 2916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:54.339229 2916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:54.339339 2916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:54.339402 2916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:54.349679 2916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:36255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:36255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:54.351083 2916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:54.352483 2916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:54.365025 2922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:54.365079 2925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:54.365149 2923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:54.365777 2916 server_base.cc:1034] running on GCE node
I20260928 15:49:54.366500 2916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:54.367856 2916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:54.369153 2916 hybrid_clock.cc:648] HybridClock initialized: now 1790610594369102 us; error 47 us; skew 500 ppm
I20260928 15:49:54.369520 2916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:54 dist-test-slave-j968 krb5kdc[2903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610594, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:54.372699 2916 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:49:54.374148 2916 webserver.cc:460] Webserver started at http://127.13.5.126:44515/ using document root <none> and password file <none>
I20260928 15:49:54.374819 2916 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:54.374920 2916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:54.375197 2916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:54.378298 2916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "710b6f4723d44e788a9a51b8f41bd674"
format_stamp: "Formatted at 2026-09-28 15:49:54 on dist-test-slave-j968"
server_key: "7a84bacf4fbf9491b6c1b7136693ee47"
server_key_iv: "e89197528b55b48dbd0cb7e5e6404074"
server_key_version: "encryptionkey@0"
I20260928 15:49:54.379019 2916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "710b6f4723d44e788a9a51b8f41bd674"
format_stamp: "Formatted at 2026-09-28 15:49:54 on dist-test-slave-j968"
server_key: "7a84bacf4fbf9491b6c1b7136693ee47"
server_key_iv: "e89197528b55b48dbd0cb7e5e6404074"
server_key_version: "encryptionkey@0"
I20260928 15:49:54.384697 2916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 15:49:54.388263 2932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:54.389845 2916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:49:54.390028 2916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "710b6f4723d44e788a9a51b8f41bd674"
format_stamp: "Formatted at 2026-09-28 15:49:54 on dist-test-slave-j968"
server_key: "7a84bacf4fbf9491b6c1b7136693ee47"
server_key_iv: "e89197528b55b48dbd0cb7e5e6404074"
server_key_version: "encryptionkey@0"
I20260928 15:49:54.390244 2916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:54.396286 2916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:54.400786 2916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:54.401130 2916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:54.420298 2916 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:36255
I20260928 15:49:54.420486 2983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:36255 every 8 connection(s)
I20260928 15:49:54.421823 2916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:54.425836 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:54.426267 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 2916
I20260928 15:49:54.426458 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:54.426923 13333 external_mini_cluster.cc:1428] Setting key 50ae90e56595bebb9ceb9d394cb9c46d
I20260928 15:49:54.436544 2987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:54.428792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43543 (local address 127.13.5.126:36255)
0928 15:49:54.429630 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:54.429646 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:54.429698 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:49:54.432650 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:54.432678 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:54.433151 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:54.433619 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:54.434637 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:54.434645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:54.434663 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:54.434678 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:54.434747 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:54.434771 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:54.434981 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:54.435118 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 15:49:54.435303 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":337,"threads_started":1}
I20260928 15:49:54.440282 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:54.440434 2984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674: Bootstrap starting.
I20260928 15:49:54.443553 2984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:54.444774 2984 log.cc:824] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:54.447309 2987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:54.444768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44901 (local address 127.13.5.126:36255)
0928 15:49:54.444957 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:54.444962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:54.445268 (+ 306us) server_negotiation.cc:407] Connection header received
0928 15:49:54.445462 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:54.445469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:54.445537 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:54.445618 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:54.446631 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:54.446636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:54.446640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:54.446658 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:54.446706 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:54.446741 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:54.446855 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:54.446970 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 15:49:54.447036 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:49:54.448202 2984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674: No bootstrap required, opened a new log
I20260928 15:49:54.455968 2984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710b6f4723d44e788a9a51b8f41bd674" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36255 } }
I20260928 15:49:54.456355 2984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:54.456473 2984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710b6f4723d44e788a9a51b8f41bd674, State: Initialized, Role: FOLLOWER
I20260928 15:49:54.457046 2984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710b6f4723d44e788a9a51b8f41bd674" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36255 } }
I20260928 15:49:54.457331 2984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:54.457480 2984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:54.457660 2984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:54.460335 2984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710b6f4723d44e788a9a51b8f41bd674" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36255 } }
I20260928 15:49:54.460846 2984 leader_election.cc:302] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 710b6f4723d44e788a9a51b8f41bd674; no voters:
I20260928 15:49:54.461464 2984 leader_election.cc:288] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:54.461603 2994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:54.462536 2994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [term 1 LEADER]: Becoming Leader. State: Replica: 710b6f4723d44e788a9a51b8f41bd674, State: Running, Role: LEADER
I20260928 15:49:54.463063 2994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710b6f4723d44e788a9a51b8f41bd674" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36255 } }
I20260928 15:49:54.464048 2984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:54.466460 2996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 710b6f4723d44e788a9a51b8f41bd674. Latest consensus state: current_term: 1 leader_uuid: "710b6f4723d44e788a9a51b8f41bd674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710b6f4723d44e788a9a51b8f41bd674" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36255 } } }
I20260928 15:49:54.466905 2996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:54.466622 2995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "710b6f4723d44e788a9a51b8f41bd674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710b6f4723d44e788a9a51b8f41bd674" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 36255 } } }
I20260928 15:49:54.467060 2995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:54.468850 3001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:54.475517 3001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:54.485029 3001 catalog_manager.cc:1347] Generated new cluster ID: c9cd9895bb3f4dcf9104e457f802e553
I20260928 15:49:54.485172 3001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:54.499079 3001 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:54.499635 3001 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:54.510687 3001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 710b6f4723d44e788a9a51b8f41bd674: Generated new TSK 0
I20260928 15:49:54.511545 3001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:54.522709 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2916
W20260928 15:49:54.524184 2979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43543
W20260928 15:49:54.524925 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 2916: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:54.525102 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 2916
2026-09-28T15:49:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:54 dist-test-slave-j968 krb5kdc[3015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:54 dist-test-slave-j968 krb5kdc[3015](info): set up 2 sockets
Sep 28 15:49:54 dist-test-slave-j968 krb5kdc[3015](info): commencing operation
krb5kdc: starting...
W20260928 15:49:55.642932 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:55 dist-test-slave-j968 krb5kdc[3015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:55.916265 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:34165
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:56.261611 3028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:56.261904 3028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:56.262012 3028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:56.269332 3028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:56.269454 3028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:56.269564 3028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:56.269670 3028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:56.269745 3028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:56.281091 3028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:34165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:34165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:56.282487 3028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:56.283854 3028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:56.296271 3033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:56.296315 3036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:56.296345 3034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:56.297154 3028 server_base.cc:1034] running on GCE node
I20260928 15:49:56.297959 3028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:56.299463 3028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:56.300693 3028 hybrid_clock.cc:648] HybridClock initialized: now 1790610596300622 us; error 57 us; skew 500 ppm
I20260928 15:49:56.301087 3028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:56 dist-test-slave-j968 krb5kdc[3015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:56.304680 3028 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:49:56.306227 3028 webserver.cc:460] Webserver started at http://127.13.5.126:46035/ using document root <none> and password file <none>
I20260928 15:49:56.306991 3028 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:56.307101 3028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:56.307371 3028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:56.310708 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "2ccc37480ebf4d9e909e80fa7b54b172"
format_stamp: "Formatted at 2026-09-28 15:49:56 on dist-test-slave-j968"
server_key: "c5666f8018c4fb2c02fc2d6ac8a9132b"
server_key_iv: "d7f4a23e5f333dbbea6c9facc946e7ed"
server_key_version: "encryptionkey@0"
I20260928 15:49:56.311600 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "2ccc37480ebf4d9e909e80fa7b54b172"
format_stamp: "Formatted at 2026-09-28 15:49:56 on dist-test-slave-j968"
server_key: "c5666f8018c4fb2c02fc2d6ac8a9132b"
server_key_iv: "d7f4a23e5f333dbbea6c9facc946e7ed"
server_key_version: "encryptionkey@0"
I20260928 15:49:56.317436 3028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:49:56.321242 3043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:56.322739 3028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:49:56.322891 3028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "2ccc37480ebf4d9e909e80fa7b54b172"
format_stamp: "Formatted at 2026-09-28 15:49:56 on dist-test-slave-j968"
server_key: "c5666f8018c4fb2c02fc2d6ac8a9132b"
server_key_iv: "d7f4a23e5f333dbbea6c9facc946e7ed"
server_key_version: "encryptionkey@0"
I20260928 15:49:56.323112 3028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:56.350713 3028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:56.355505 3028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:56.355749 3028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:56.374728 3028 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:34165
I20260928 15:49:56.374764 3095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:34165 every 8 connection(s)
I20260928 15:49:56.376194 3028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:56.380038 3096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:56.383039 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3028
I20260928 15:49:56.383239 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:56.383677 13333 external_mini_cluster.cc:1428] Setting key ef4c45aa32eed10628d60740e2833901
I20260928 15:49:56.393345 3096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172: Bootstrap starting.
I20260928 15:49:56.397162 3096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:56.398226 3096 log.cc:824] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:49:56 dist-test-slave-j968 krb5kdc[3015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:49:56.401619 3096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172: No bootstrap required, opened a new log
I20260928 15:49:56.409735 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:56.385728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60019 (local address 127.13.5.126:34165)
0928 15:49:56.386492 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:56.386508 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:56.386566 (+ 58us) server_negotiation.cc:407] Connection header received
0928 15:49:56.388918 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:56.388949 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:56.389501 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:56.389967 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:56.391216 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:56.393209 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:56.394466 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:56.395594 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:56.395782 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:56.400512 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:56.400523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:56.400542 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:56.400601 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:56.404817 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:56.405546 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:56.405561 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:56.405571 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:56.405800 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:56.406191 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:56.406197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:56.406202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:56.406960 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:56.407261 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:56.407714 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 15:49:56.408238 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":308,"threads_started":1}
I20260928 15:49:56.412654 3096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc37480ebf4d9e909e80fa7b54b172" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34165 } }
I20260928 15:49:56.413070 3096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:56.413195 3096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccc37480ebf4d9e909e80fa7b54b172, State: Initialized, Role: FOLLOWER
I20260928 15:49:56.413839 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:56.413873 3096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc37480ebf4d9e909e80fa7b54b172" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34165 } }
I20260928 15:49:56.414160 3096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:56.414302 3096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:56.414469 3096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:56.416728 3096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc37480ebf4d9e909e80fa7b54b172" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34165 } }
I20260928 15:49:56.417239 3096 leader_election.cc:302] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ccc37480ebf4d9e909e80fa7b54b172; no voters:
I20260928 15:49:56.417809 3096 leader_election.cc:288] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:56.417949 3105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:56.419857 3096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:56.420737 3105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [term 1 LEADER]: Becoming Leader. State: Replica: 2ccc37480ebf4d9e909e80fa7b54b172, State: Running, Role: LEADER
I20260928 15:49:56.421399 3105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc37480ebf4d9e909e80fa7b54b172" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34165 } }
I20260928 15:49:56.424616 3107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ccc37480ebf4d9e909e80fa7b54b172. Latest consensus state: current_term: 1 leader_uuid: "2ccc37480ebf4d9e909e80fa7b54b172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc37480ebf4d9e909e80fa7b54b172" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34165 } } }
I20260928 15:49:56.425014 3107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:56.425576 3106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ccc37480ebf4d9e909e80fa7b54b172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccc37480ebf4d9e909e80fa7b54b172" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 34165 } } }
I20260928 15:49:56.425879 3106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172 [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:56.427657 3116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:56.435999 3116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:56.440809 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:56.418907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34203 (local address 127.13.5.126:34165)
0928 15:49:56.419067 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:56.419075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:56.420634 (+ 1559us) server_negotiation.cc:407] Connection header received
0928 15:49:56.420920 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:56.420926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:56.421013 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:56.421086 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:56.423428 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:56.424977 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:56.427575 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:56.428577 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:56.428769 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:56.432718 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:56.432726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:56.432743 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:49:56.432787 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:56.437109 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:56.438094 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:56.438099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:56.438103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:56.438253 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:49:56.438770 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:49:56.438775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:49:56.438778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:49:56.439205 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:56.439441 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:56.440130 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 15:49:56.440423 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:49:56.446462 3116 catalog_manager.cc:1347] Generated new cluster ID: 355cb38e2507431f94fa205a18859c3b
I20260928 15:49:56.446590 3116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:56.481498 3116 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:56.483472 3116 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:56.494766 3116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ccc37480ebf4d9e909e80fa7b54b172: Generated new TSK 0
I20260928 15:49:56.495901 3116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:56.503338 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3028
I20260928 15:49:56.504544 3091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:49:56.600769 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3028
2026-09-28T15:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:56 dist-test-slave-j968 krb5kdc[3129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:56 dist-test-slave-j968 krb5kdc[3129](info): set up 2 sockets
Sep 28 15:49:56 dist-test-slave-j968 krb5kdc[3129](info): commencing operation
krb5kdc: starting...
W20260928 15:49:57.718894 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:57 dist-test-slave-j968 krb5kdc[3129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:57.995165 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:41723
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:49:58.342115 3142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:49:58.342502 3142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:49:58.342654 3142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:49:58.350243 3142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:49:58.350425 3142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:49:58.350524 3142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:49:58.350625 3142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:49:58.350696 3142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:49:58.361099 3142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:49:58.362474 3142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:49:58.363910 3142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:49:58.375844 3150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:58.375983 3147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:49:58.375983 3148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:49:58.376890 3142 server_base.cc:1034] running on GCE node
I20260928 15:49:58.377754 3142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:49:58.379382 3142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:49:58.380605 3142 hybrid_clock.cc:648] HybridClock initialized: now 1790610598380542 us; error 60 us; skew 500 ppm
I20260928 15:49:58.380961 3142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:49:58 dist-test-slave-j968 krb5kdc[3129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610598, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:49:58.384270 3142 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:49:58.385807 3142 webserver.cc:460] Webserver started at http://127.13.5.126:32845/ using document root <none> and password file <none>
I20260928 15:49:58.386528 3142 fs_manager.cc:359] Metadata directory not provided
I20260928 15:49:58.386669 3142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:49:58.386977 3142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:49:58.390123 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "048411840eeb43ca8cfeddc4b106287a"
format_stamp: "Formatted at 2026-09-28 15:49:58 on dist-test-slave-j968"
server_key: "111319ab7cd487d32a9ef705d9230b38"
server_key_iv: "0c46f18f51e23103272f7e82e93d2718"
server_key_version: "encryptionkey@0"
I20260928 15:49:58.390910 3142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "048411840eeb43ca8cfeddc4b106287a"
format_stamp: "Formatted at 2026-09-28 15:49:58 on dist-test-slave-j968"
server_key: "111319ab7cd487d32a9ef705d9230b38"
server_key_iv: "0c46f18f51e23103272f7e82e93d2718"
server_key_version: "encryptionkey@0"
I20260928 15:49:58.396430 3142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:49:58.400310 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:49:58.401806 3142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:49:58.402024 3142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "048411840eeb43ca8cfeddc4b106287a"
format_stamp: "Formatted at 2026-09-28 15:49:58 on dist-test-slave-j968"
server_key: "111319ab7cd487d32a9ef705d9230b38"
server_key_iv: "0c46f18f51e23103272f7e82e93d2718"
server_key_version: "encryptionkey@0"
I20260928 15:49:58.402251 3142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:49:58.425055 3142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:49:58.429353 3142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:49:58.429657 3142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:49:58.448680 3142 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:41723
I20260928 15:49:58.448726 3209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:41723 every 8 connection(s)
I20260928 15:49:58.450107 3142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:49:58.453896 3210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:49:58.459012 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3142
I20260928 15:49:58.459223 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:49:58.459826 13333 external_mini_cluster.cc:1428] Setting key 3b39338156feadf900b4dd2ff3092112
I20260928 15:49:58.465551 3210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a: Bootstrap starting.
I20260928 15:49:58.468791 3210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a: Neither blocks nor log segments found. Creating new log.
I20260928 15:49:58.469870 3210 log.cc:824] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:49:58.473460 3210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a: No bootstrap required, opened a new log
I20260928 15:49:58.473939 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:58.461814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37787 (local address 127.13.5.126:41723)
0928 15:49:58.462634 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:58.462646 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:58.462689 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:49:58.464505 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:58.464527 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:58.464931 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:58.465316 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:58.465846 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:58.467162 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:58.468791 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:58.469576 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:58.469790 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:58.471124 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:58.471134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:58.471149 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:58.471903 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:58.471995 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:58.472027 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:58.472156 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:58.472376 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 15:49:58.472652 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":369,"threads_started":1}
I20260928 15:49:58.477836 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:49:58.482021 3210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "048411840eeb43ca8cfeddc4b106287a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41723 } }
I20260928 15:49:58.482560 3210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:49:58.482733 3210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048411840eeb43ca8cfeddc4b106287a, State: Initialized, Role: FOLLOWER
I20260928 15:49:58.483459 3210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "048411840eeb43ca8cfeddc4b106287a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41723 } }
I20260928 15:49:58.483759 3210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:49:58.483942 3210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:49:58.484122 3210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:49:58.486894 3210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "048411840eeb43ca8cfeddc4b106287a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41723 } }
I20260928 15:49:58.487521 3210 leader_election.cc:302] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 048411840eeb43ca8cfeddc4b106287a; no voters:
I20260928 15:49:58.488200 3210 leader_election.cc:288] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:49:58.488401 3220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:49:58.489946 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:49:58.482446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36001 (local address 127.13.5.126:41723)
0928 15:49:58.482640 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:49:58.482648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:49:58.483171 (+ 523us) server_negotiation.cc:407] Connection header received
0928 15:49:58.483351 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:49:58.483356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:49:58.483449 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:49:58.483566 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:49:58.484201 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:58.485331 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:58.486719 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:49:58.487380 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:49:58.487528 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:49:58.489038 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:49:58.489047 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:49:58.489055 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:49:58.489085 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:49:58.489148 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:49:58.489176 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:49:58.489257 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:49:58.489519 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 15:49:58.489620 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:49:58.490609 3210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:49:58.492635 3220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [term 1 LEADER]: Becoming Leader. State: Replica: 048411840eeb43ca8cfeddc4b106287a, State: Running, Role: LEADER
I20260928 15:49:58.493211 3220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "048411840eeb43ca8cfeddc4b106287a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41723 } }
I20260928 15:49:58.495785 3222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 048411840eeb43ca8cfeddc4b106287a. Latest consensus state: current_term: 1 leader_uuid: "048411840eeb43ca8cfeddc4b106287a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "048411840eeb43ca8cfeddc4b106287a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41723 } } }
I20260928 15:49:58.496220 3222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:58.496163 3221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "048411840eeb43ca8cfeddc4b106287a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "048411840eeb43ca8cfeddc4b106287a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 41723 } } }
I20260928 15:49:58.496536 3221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a [sys.catalog]: This master's current role is: LEADER
I20260928 15:49:58.499567 3231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:49:58.503969 3231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:49:58.512614 3231 catalog_manager.cc:1347] Generated new cluster ID: 00edfb37a62c43c2bb6133bbefcc2b69
I20260928 15:49:58.512763 3231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:49:58.531188 3231 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:49:58.533187 3231 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:49:58.549062 3231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 048411840eeb43ca8cfeddc4b106287a: Generated new TSK 0
I20260928 15:49:58.550433 3231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:49:58.558637 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3142
W20260928 15:49:58.560277 3205 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37787
W20260928 15:49:58.561018 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3142: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:49:58.561177 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3142
2026-09-28T15:49:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:49:58 dist-test-slave-j968 krb5kdc[3241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:49:58 dist-test-slave-j968 krb5kdc[3241](info): set up 2 sockets
Sep 28 15:49:58 dist-test-slave-j968 krb5kdc[3241](info): commencing operation
krb5kdc: starting...
W20260928 15:49:59.690977 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:49:59 dist-test-slave-j968 krb5kdc[3241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:49:59.971366 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:33563
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:33563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:00.322161 3254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:00.322482 3254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:00.322649 3254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:00.330156 3254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:00.330281 3254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:00.330392 3254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:00.330494 3254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:00.330575 3254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:00.341104 3254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:33563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:33563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:00.342607 3254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:00.344035 3254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:00.356505 3259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:00.356539 3260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:00.356709 3262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:00.357982 3254 server_base.cc:1034] running on GCE node
I20260928 15:50:00.358829 3254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:00.360421 3254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:00.361636 3254 hybrid_clock.cc:648] HybridClock initialized: now 1790610600361596 us; error 55 us; skew 500 ppm
I20260928 15:50:00.362000 3254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:00 dist-test-slave-j968 krb5kdc[3241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610600, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:00.365296 3254 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:50:00.366899 3254 webserver.cc:460] Webserver started at http://127.13.5.126:46669/ using document root <none> and password file <none>
I20260928 15:50:00.367635 3254 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:00.367781 3254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:00.368153 3254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:00.371335 3254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "6754c21533844b93a9e9fa3565aff96a"
format_stamp: "Formatted at 2026-09-28 15:50:00 on dist-test-slave-j968"
server_key: "6bc3236c8b19502bf2ef78a263f0be90"
server_key_iv: "1b6bef6dda4d01e6347e1cfa022d3aa3"
server_key_version: "encryptionkey@0"
I20260928 15:50:00.372213 3254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "6754c21533844b93a9e9fa3565aff96a"
format_stamp: "Formatted at 2026-09-28 15:50:00 on dist-test-slave-j968"
server_key: "6bc3236c8b19502bf2ef78a263f0be90"
server_key_iv: "1b6bef6dda4d01e6347e1cfa022d3aa3"
server_key_version: "encryptionkey@0"
I20260928 15:50:00.377712 3254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:50:00.381443 3269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:00.382930 3254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:50:00.383149 3254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "6754c21533844b93a9e9fa3565aff96a"
format_stamp: "Formatted at 2026-09-28 15:50:00 on dist-test-slave-j968"
server_key: "6bc3236c8b19502bf2ef78a263f0be90"
server_key_iv: "1b6bef6dda4d01e6347e1cfa022d3aa3"
server_key_version: "encryptionkey@0"
I20260928 15:50:00.383379 3254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:00.389570 3254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:00.394233 3254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:00.394544 3254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:00.414238 3254 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:33563
I20260928 15:50:00.414276 3320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:33563 every 8 connection(s)
I20260928 15:50:00.415793 3254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:00.418381 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3254
I20260928 15:50:00.418582 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:50:00.419117 13333 external_mini_cluster.cc:1428] Setting key 41e90946a1337a01d8c5528849da94ba
I20260928 15:50:00.420302 3321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:00.427006 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:00.420998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40857 (local address 127.13.5.126:33563)
0928 15:50:00.421827 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:00.421839 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:00.421881 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:50:00.423474 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:00.423497 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:00.423860 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:00.424320 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:00.425386 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:00.425393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:00.425406 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:50:00.425420 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:00.425496 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:50:00.425515 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:00.425698 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:00.425816 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 15:50:00.426039 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":372,"threads_started":1}
I20260928 15:50:00.430755 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:50:00.434182 3321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a: Bootstrap starting.
I20260928 15:50:00.438409 3321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:00.439014 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:00.436233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39007 (local address 127.13.5.126:33563)
0928 15:50:00.436451 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:00.436459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:00.436991 (+ 532us) server_negotiation.cc:407] Connection header received
0928 15:50:00.437165 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:00.437168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:00.437217 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:00.437298 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:00.438322 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:00.438326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:00.438330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:50:00.438344 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:00.438397 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:50:00.438411 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:00.438534 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:00.438660 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 15:50:00.438709 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:50:00.439786 3321 log.cc:824] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:00.442672 3321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a: No bootstrap required, opened a new log
I20260928 15:50:00.450006 3321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6754c21533844b93a9e9fa3565aff96a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33563 } }
I20260928 15:50:00.450428 3321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:00.450542 3321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6754c21533844b93a9e9fa3565aff96a, State: Initialized, Role: FOLLOWER
I20260928 15:50:00.451134 3321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6754c21533844b93a9e9fa3565aff96a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33563 } }
I20260928 15:50:00.451367 3321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:00.451517 3321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:00.451690 3321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:00.453962 3321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6754c21533844b93a9e9fa3565aff96a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33563 } }
I20260928 15:50:00.454475 3321 leader_election.cc:302] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6754c21533844b93a9e9fa3565aff96a; no voters:
I20260928 15:50:00.455112 3321 leader_election.cc:288] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:00.455259 3331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:00.455924 3331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [term 1 LEADER]: Becoming Leader. State: Replica: 6754c21533844b93a9e9fa3565aff96a, State: Running, Role: LEADER
I20260928 15:50:00.456528 3331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6754c21533844b93a9e9fa3565aff96a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33563 } }
I20260928 15:50:00.457453 3321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:00.459672 3333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6754c21533844b93a9e9fa3565aff96a. Latest consensus state: current_term: 1 leader_uuid: "6754c21533844b93a9e9fa3565aff96a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6754c21533844b93a9e9fa3565aff96a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33563 } } }
I20260928 15:50:00.460181 3333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:00.460088 3332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6754c21533844b93a9e9fa3565aff96a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6754c21533844b93a9e9fa3565aff96a" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 33563 } } }
I20260928 15:50:00.460391 3332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:00.462028 3338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:00.467759 3338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:00.477792 3338 catalog_manager.cc:1347] Generated new cluster ID: 09e957ecb70946a39b56db3d66b35ccf
I20260928 15:50:00.477934 3338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:00.491784 3338 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:00.492326 3338 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:00.511512 3338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6754c21533844b93a9e9fa3565aff96a: Generated new TSK 0
I20260928 15:50:00.512753 3338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:50:00.518575 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3254
W20260928 15:50:00.519764 3316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40857
W20260928 15:50:00.520567 13333 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:00.520731 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3254
2026-09-28T15:50:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1958 ms)
[----------] 18 tests from AuthTokenIssuingTest (36037 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:00 dist-test-slave-j968 krb5kdc[3352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:00 dist-test-slave-j968 krb5kdc[3352](info): set up 2 sockets
Sep 28 15:50:00 dist-test-slave-j968 krb5kdc[3352](info): commencing operation
krb5kdc: starting...
W20260928 15:50:01.638962 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:50:01 dist-test-slave-j968 krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:50:01.920274 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43855
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43855,127.13.5.125:40765,127.13.5.124:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:02.270040 3366 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:50:02.270401 3366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:02.270530 3366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:02.270673 3366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:02.278133 3366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:02.278234 3366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:02.278331 3366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:02.278435 3366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:02.278545 3366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:02.288965 3366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43855,127.13.5.125:40765,127.13.5.124:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:43855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:02.290380 3366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:02.291787 3366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:02.306896 3371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:02.306885 3372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:02.307278 3374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:02.307843 3366 server_base.cc:1034] running on GCE node
I20260928 15:50:02.308699 3366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:02.310175 3366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:02.311496 3366 hybrid_clock.cc:648] HybridClock initialized: now 1790610602311292 us; error 116 us; skew 500 ppm
I20260928 15:50:02.311833 3366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:02 dist-test-slave-j968 krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610602, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:02.315313 3366 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:50:02.317122 3366 webserver.cc:460] Webserver started at http://127.13.5.126:37721/ using document root <none> and password file <none>
I20260928 15:50:02.317830 3366 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:02.317920 3366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:02.318236 3366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:02.321462 3366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "380e36f52a4a48a2863de8a3c1f02616"
format_stamp: "Formatted at 2026-09-28 15:50:02 on dist-test-slave-j968"
server_key: "7281a1392a8db9ac9d0d015605e1d919"
server_key_iv: "c7eb40979f151375a637fbfd9bf68e4a"
server_key_version: "encryptionkey@0"
I20260928 15:50:02.322243 3366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "380e36f52a4a48a2863de8a3c1f02616"
format_stamp: "Formatted at 2026-09-28 15:50:02 on dist-test-slave-j968"
server_key: "7281a1392a8db9ac9d0d015605e1d919"
server_key_iv: "c7eb40979f151375a637fbfd9bf68e4a"
server_key_version: "encryptionkey@0"
I20260928 15:50:02.328231 3366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:50:02.331727 3381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:02.333262 3366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:02.333464 3366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "380e36f52a4a48a2863de8a3c1f02616"
format_stamp: "Formatted at 2026-09-28 15:50:02 on dist-test-slave-j968"
server_key: "7281a1392a8db9ac9d0d015605e1d919"
server_key_iv: "c7eb40979f151375a637fbfd9bf68e4a"
server_key_version: "encryptionkey@0"
I20260928 15:50:02.333695 3366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:02.373324 3366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:02.378173 3366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:02.378479 3366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:02.397734 3366 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:43855
I20260928 15:50:02.397848 3432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:43855 every 8 connection(s)
I20260928 15:50:02.399247 3366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:02.403152 3433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:02.408341 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3366
I20260928 15:50:02.408573 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:50:02.409179 13333 external_mini_cluster.cc:1428] Setting key 58ab8b1300a79386b7272b7c2fcbf333
I20260928 15:50:02.410220 3433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:50:02 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610602, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.125@KRBTEST.COM; defaulting to no policy
I20260928 15:50:02.433833 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.413712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.13.5.126:43855)
0928 15:50:02.414583 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:02.414604 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:02.415082 (+ 478us) server_negotiation.cc:407] Connection header received
0928 15:50:02.417289 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:02.417295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:02.417530 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:02.417759 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:02.419056 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.420345 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.421742 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.422298 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.422516 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:02.425924 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:02.425933 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:02.425947 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:02.425982 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:02.428871 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:02.429593 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:02.429601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:02.429603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:02.429751 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:02.430366 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:02.430372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:02.430379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:02.430955 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:02.431226 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:02.431787 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 15:50:02.432258 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":594,"thread_start_us":372,"threads_started":1}
I20260928 15:50:02.433851 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.413712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43855 (local address 127.0.0.1:54216)
0928 15:50:02.414896 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.414932 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:02.416887 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:02.417760 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:02.417767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:02.418508 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:02.418844 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:02.418861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.420495 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.420500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:02.421600 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:02.421613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.422549 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.422556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:02.422725 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:02.423393 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:02.423427 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:02.425707 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:02.429019 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:02.429033 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:02.429048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:02.429450 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:02.429860 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:02.429866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:02.429870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:02.430035 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:02.431100 (+ 1065us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:02.431108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:02.431376 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 15:50:02.431658 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 15:50:02.432375 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":211,"thread_start_us":728,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.125@KRBTEST.COM" created.
I20260928 15:50:02.437314 3433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:02.438730 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.438203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60490)
0928 15:50:02.438412 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.438519 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:50:02.439009 3385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
I20260928 15:50:02.440652 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.440301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60498)
0928 15:50:02.440449 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.440523 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:50:02.440876 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:40765: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab.
Entry for principal kudu/127.13.5.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab.
I20260928 15:50:02.468680 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.125:40765
--webserver_interface=127.13.5.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab
--principal=kudu/127.13.5.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43855,127.13.5.125:40765,127.13.5.124:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:50:02.490229 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } attempt: 1
I20260928 15:50:02.491582 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.490936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60512)
0928 15:50:02.491220 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.491352 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 15:50:02.492816 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.492329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60528)
0928 15:50:02.492514 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.492616 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:50:02.493041 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:40765: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
I20260928 15:50:02.561353 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } attempt: 2
I20260928 15:50:02.562826 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.562221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60534)
0928 15:50:02.562523 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.562642 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:50:02.564139 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.563590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60538)
0928 15:50:02.563778 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.563940 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:50:02.564365 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:40765: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
I20260928 15:50:02.655668 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } attempt: 3
I20260928 15:50:02.657023 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.656464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60548)
0928 15:50:02.656720 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.656839 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 15:50:02.658126 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.657713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60556)
0928 15:50:02.657864 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.657951 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:50:02.658320 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:40765: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
I20260928 15:50:02.801589 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } attempt: 4
I20260928 15:50:02.803085 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.802335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60564)
0928 15:50:02.802759 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.802885 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260928 15:50:02.804325 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.803780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60572)
0928 15:50:02.804040 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.804155 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:50:02.804605 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:40765: Network error: Client connection negotiation failed: client connection to 127.13.5.125:40765: connect: Connection refused (error 111)
W20260928 15:50:02.834161 3440 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:50:02.834591 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:02.834723 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:02.834868 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:02.843245 3440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:02.843394 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:02.843516 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:02.843600 3440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:02.843700 3440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:02.854084 3440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43855,127.13.5.125:40765,127.13.5.124:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab
--principal=kudu/127.13.5.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.125:40765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.5.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:02.855535 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:02.856959 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:02.869259 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:02.869228 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:02.869410 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:02.870098 3440 server_base.cc:1034] running on GCE node
I20260928 15:50:02.870851 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:02.872351 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:02.873615 3440 hybrid_clock.cc:648] HybridClock initialized: now 1790610602873549 us; error 30 us; skew 500 ppm
I20260928 15:50:02.873981 3440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:02 dist-test-slave-j968 krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610602, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:02.877421 3440 init.cc:377] Logged in from keytab as kudu/127.13.5.125@KRBTEST.COM (short username kudu)
I20260928 15:50:02.878821 3440 webserver.cc:460] Webserver started at http://127.13.5.125:39503/ using document root <none> and password file <none>
I20260928 15:50:02.879488 3440 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:02.879578 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:02.879839 3440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:02.882848 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/data/instance:
uuid: "bdd5e7ab2de345bb87953049895c4793"
format_stamp: "Formatted at 2026-09-28 15:50:02 on dist-test-slave-j968"
server_key: "8f47dfb1810b3f5eab9d2fb885b2d266"
server_key_iv: "06d187bb0c95ab3e50d5ca9ec4bc7eb0"
server_key_version: "encryptionkey@0"
I20260928 15:50:02.883608 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/wal/instance:
uuid: "bdd5e7ab2de345bb87953049895c4793"
format_stamp: "Formatted at 2026-09-28 15:50:02 on dist-test-slave-j968"
server_key: "8f47dfb1810b3f5eab9d2fb885b2d266"
server_key_iv: "06d187bb0c95ab3e50d5ca9ec4bc7eb0"
server_key_version: "encryptionkey@0"
I20260928 15:50:02.888965 3440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 15:50:02.892480 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:02.893869 3440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:50:02.894055 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/wal
uuid: "bdd5e7ab2de345bb87953049895c4793"
format_stamp: "Formatted at 2026-09-28 15:50:02 on dist-test-slave-j968"
server_key: "8f47dfb1810b3f5eab9d2fb885b2d266"
server_key_iv: "06d187bb0c95ab3e50d5ca9ec4bc7eb0"
server_key_version: "encryptionkey@0"
I20260928 15:50:02.894279 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:02.920753 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:02.925338 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:02.925628 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:02.944873 3440 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.125:40765
I20260928 15:50:02.944893 3506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.125:40765 every 8 connection(s)
I20260928 15:50:02.946244 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
I20260928 15:50:02.946658 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3440
I20260928 15:50:02.946861 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-1/wal/instance
I20260928 15:50:02.947456 13333 external_mini_cluster.cc:1428] Setting key a56df59bab21157481b70592af98f84c
I20260928 15:50:02.950358 3507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:02.957197 3507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.5.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.124@KRBTEST.COM" created.
Sep 28 15:50:02 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610602, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:50:02.983021 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.959771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.13.5.126:43855)
0928 15:50:02.960288 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:02.960295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:02.961408 (+ 1113us) server_negotiation.cc:407] Connection header received
0928 15:50:02.964355 (+ 2947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:02.964359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:02.964431 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:02.964528 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:02.966781 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.967511 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.970188 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.970629 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.970734 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:02.977217 (+ 6483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:02.977223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:02.977227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:02.977274 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:02.979942 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:02.980677 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:02.980682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:02.980689 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:02.980809 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:02.981229 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:02.981234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:02.981237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:02.981535 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:02.981704 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:02.982405 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 15:50:02.982629 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
I20260928 15:50:02.984117 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.960380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43855 (local address 127.0.0.1:54226)
0928 15:50:02.961197 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.961258 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:02.964046 (+ 2788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:02.964817 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:02.964831 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:02.965645 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:02.966527 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:02.966563 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.967691 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.967699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:02.970013 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:02.970033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.970887 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.970895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:02.971248 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:02.972026 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:02.972066 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:02.976954 (+ 4888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:02.980085 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:02.980094 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:02.980112 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:02.980539 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:02.980916 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:02.980921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:02.980925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:02.981111 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:02.981666 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:02.981675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:02.982038 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 15:50:02.982302 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 15:50:02.982862 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":329,"threads_started":1}
I20260928 15:50:02.985839 3507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:50:02 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610602, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:50:03.002800 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.986600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60582)
0928 15:50:02.986787 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:02.986808 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:02.986907 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:02.988145 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:02.988148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:02.988480 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:02.988614 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:02.988622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.990632 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.990636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:02.991469 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:02.991481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.992731 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.992735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:02.992855 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:02.993472 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:02.993498 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:02.996026 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:03.000411 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.000415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.000417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.000772 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.001198 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.001201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.001204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.001320 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.001986 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:03.001989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:03.002113 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 15:50:03.002223 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:50:03.002508 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 15:50:03.003234 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:02.986883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.13.5.125:40765)
0928 15:50:02.987593 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:02.987604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:02.987635 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:50:02.987691 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:02.987701 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:02.987964 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:02.988111 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:02.988783 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.990500 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.991612 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:02.992599 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:02.992736 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:02.996245 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:02.996262 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:02.996282 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:02.996326 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.000290 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.000912 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.000923 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.000927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.001075 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.001444 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.001450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.001453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.001871 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.002037 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.002566 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.002783 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":367,"threads_started":1}
I20260928 15:50:03.005726 3507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:03.006868 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.006414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55522)
0928 15:50:03.006587 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.006700 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 15:50:03.007146 3456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
I20260928 15:50:03.008843 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.008508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55532)
0928 15:50:03.008647 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.008745 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 15:50:03.009094 3507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab.
Entry for principal kudu/127.13.5.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab.
I20260928 15:50:03.011390 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.124:43575
--webserver_interface=127.13.5.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab
--principal=kudu/127.13.5.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:42005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:43855,127.13.5.125:40765,127.13.5.124:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:50:03.058442 3507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 1
I20260928 15:50:03.059892 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.059248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55534)
0928 15:50:03.059524 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.059662 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 15:50:03.060973 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.060641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55542)
0928 15:50:03.060784 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.060872 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:50:03.061270 3507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
I20260928 15:50:03.103950 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } attempt: 5
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610602, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:50:03.119463 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.104711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60590)
0928 15:50:03.104974 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.105001 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:03.105176 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:03.105558 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:03.105564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:03.106082 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:03.106330 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.106345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.107353 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.107358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.108846 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.108870 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.109809 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.109814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.110070 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.110796 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:03.110820 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:03.113420 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:03.116783 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.116788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.116792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.117207 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.117650 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.117656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.117659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.117820 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.118432 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:03.118437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:03.118601 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 15:50:03.118744 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 15:50:03.119093 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 15:50:03.119450 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.104816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.13.5.125:40765)
0928 15:50:03.104998 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.105003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.105092 (+ 89us) server_negotiation.cc:407] Connection header received
0928 15:50:03.105305 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.105309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.105395 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.105481 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.106495 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.107207 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.109080 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.109619 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.109750 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.113767 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.113772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.113776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.113815 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.116621 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.117332 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.117338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.117341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.117493 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.117971 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.117974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.117976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.118305 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.118459 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.118851 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.119097 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:50:03.120545 3433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:03.121774 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.121253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55546)
0928 15:50:03.121551 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.121628 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 15:50:03.122829 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.122418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55548)
0928 15:50:03.122614 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.122689 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 15:50:03.123087 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
I20260928 15:50:03.129482 3507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 2
I20260928 15:50:03.130525 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.130101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55560)
0928 15:50:03.130284 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.130377 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:50:03.131592 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.131244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55566)
0928 15:50:03.131391 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.131454 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 15:50:03.131860 3507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
I20260928 15:50:03.174355 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 1
I20260928 15:50:03.175791 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.175165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55580)
0928 15:50:03.175505 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.175613 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 15:50:03.176873 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.176530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55582)
0928 15:50:03.176681 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.176765 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:50:03.177093 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
I20260928 15:50:03.223093 3507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 3
I20260928 15:50:03.224283 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.223785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55596)
0928 15:50:03.224024 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.224120 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 15:50:03.225364 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.224945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55598)
0928 15:50:03.225167 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.225252 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260928 15:50:03.225553 3507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
I20260928 15:50:03.245316 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 2
I20260928 15:50:03.246508 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.246002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55602)
0928 15:50:03.246232 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.246332 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 15:50:03.247599 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.247259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55610)
0928 15:50:03.247414 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.247491 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:50:03.247799 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
I20260928 15:50:03.354040 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 3
I20260928 15:50:03.355253 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.354746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55622)
0928 15:50:03.354996 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.355093 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 15:50:03.356323 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.355944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55632)
0928 15:50:03.356157 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.356224 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 15:50:03.356536 3433 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
W20260928 15:50:03.360415 3515 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:50:03.360774 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:03.360896 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:03.361037 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:03.368494 3515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:03.368619 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:03.368749 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:03.368834 3515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:03.368933 3515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:03.368770 3507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 4
I20260928 15:50:03.369961 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.369419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55638)
0928 15:50:03.369677 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.369785 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 15:50:03.371001 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.370629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55646)
0928 15:50:03.370791 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.370872 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:50:03.371220 3507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:43575: Network error: Client connection negotiation failed: client connection to 127.13.5.124:43575: connect: Connection refused (error 111)
I20260928 15:50:03.379782 3515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:42005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:43855,127.13.5.125:40765,127.13.5.124:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab
--principal=kudu/127.13.5.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.124:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.5.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:03.381253 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:03.382583 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:03.394632 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:03.394848 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:03.394671 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:03.399166 3515 server_base.cc:1034] running on GCE node
I20260928 15:50:03.400282 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:03.402099 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:03.403375 3515 hybrid_clock.cc:648] HybridClock initialized: now 1790610603403279 us; error 60 us; skew 500 ppm
I20260928 15:50:03.403786 3515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610603, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:03.407840 3515 init.cc:377] Logged in from keytab as kudu/127.13.5.124@KRBTEST.COM (short username kudu)
I20260928 15:50:03.409471 3515 webserver.cc:460] Webserver started at http://127.13.5.124:46201/ using document root <none> and password file <none>
I20260928 15:50:03.410274 3515 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:03.410413 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:03.410754 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:03.414093 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/data/instance:
uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8"
format_stamp: "Formatted at 2026-09-28 15:50:03 on dist-test-slave-j968"
server_key: "3086ce2da0ddbd709f459b24120525d6"
server_key_iv: "ef8336991ca7156a449b0020b2ae6090"
server_key_version: "encryptionkey@0"
I20260928 15:50:03.415156 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/wal/instance:
uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8"
format_stamp: "Formatted at 2026-09-28 15:50:03 on dist-test-slave-j968"
server_key: "3086ce2da0ddbd709f459b24120525d6"
server_key_iv: "ef8336991ca7156a449b0020b2ae6090"
server_key_version: "encryptionkey@0"
I20260928 15:50:03.421054 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 15:50:03.424647 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:03.426021 3515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:50:03.426232 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/wal
uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8"
format_stamp: "Formatted at 2026-09-28 15:50:03 on dist-test-slave-j968"
server_key: "3086ce2da0ddbd709f459b24120525d6"
server_key_iv: "ef8336991ca7156a449b0020b2ae6090"
server_key_version: "encryptionkey@0"
I20260928 15:50:03.426473 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:03.460695 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:03.465134 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:03.465409 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:03.484349 3515 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.124:43575
I20260928 15:50:03.484360 3582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.124:43575 every 8 connection(s)
I20260928 15:50:03.485754 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
I20260928 15:50:03.488197 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3515
I20260928 15:50:03.488394 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610238672534-13333-0/minicluster-data/master-2/wal/instance
I20260928 15:50:03.488994 13333 external_mini_cluster.cc:1428] Setting key 1aace4078af7975ab56fb10e382f0ffc
I20260928 15:50:03.490365 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:03.500176 3583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610603, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:50:03.513722 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.490896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.13.5.124:43575)
0928 15:50:03.491682 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.491698 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.491752 (+ 54us) server_negotiation.cc:407] Connection header received
0928 15:50:03.493797 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.493825 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.494348 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.494787 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.496201 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.498267 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.499677 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.500752 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.500960 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.505428 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.505444 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.505463 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.505508 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.509192 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.510091 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.510112 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.510132 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.510364 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.510744 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.510750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.510758 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.511474 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.511759 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.512210 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.512653 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":318,"threads_started":1}
I20260928 15:50:03.519485 3588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.503053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:43855 (local address 127.0.0.1:54236)
0928 15:50:03.503991 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.504034 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:03.504187 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:03.504614 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:03.504620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:03.505167 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:03.505570 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.505587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.506815 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.506822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.508714 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.508731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.509553 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.509558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.509677 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.510272 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:03.510299 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:03.513004 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:03.516467 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.516478 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.516484 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.517130 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.517569 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.517572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.517575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.517694 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.518239 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:03.518247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:03.518497 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 15:50:03.518697 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 15:50:03.519035 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":532,"threads_started":1}
I20260928 15:50:03.519491 3587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.503055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.13.5.126:43855)
0928 15:50:03.503637 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.503645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.504127 (+ 482us) server_negotiation.cc:407] Connection header received
0928 15:50:03.504327 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.504334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.504421 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.504544 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.505733 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.506648 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.508891 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.509371 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.509511 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.513207 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.513212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.513216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.513251 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.516018 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.517265 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.517275 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.517278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.517450 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.517820 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.517824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.517826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.518115 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.518268 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.518821 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.519035 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":240,"threads_started":1}
I20260928 15:50:03.520853 3583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:03.533731 3433 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 4
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610603, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610602, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
I20260928 15:50:03.547703 3588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.521631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:40765 (local address 127.0.0.1:60602)
0928 15:50:03.521886 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.521906 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:03.522029 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:03.522342 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:03.522347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:03.522718 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:03.522851 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.522858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.523814 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.523819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.524744 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.524755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.525440 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.525446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.525640 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.538023 (+ 12383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:03.538049 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:03.540814 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:03.544554 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.544561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.544565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.545137 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.545593 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.545604 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.545608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.545834 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.546664 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:03.546670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:03.546838 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 15:50:03.546995 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 15:50:03.547307 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260928 15:50:03.547703 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.521613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.13.5.125:40765)
0928 15:50:03.521801 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.521807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.521967 (+ 160us) server_negotiation.cc:407] Connection header received
0928 15:50:03.522162 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.522167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.522227 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.522322 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.522989 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.523668 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.524892 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.525299 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.525429 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.541040 (+ 15611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.541046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.541049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.541092 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.544426 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.545280 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.545287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.545291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.545461 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.545987 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.545992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.545996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.546406 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.546561 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.547092 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.547310 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:50:03.549269 3583 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:03.549717 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.534361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55656)
0928 15:50:03.534540 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.534555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:03.534643 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:03.534982 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:03.534986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:03.535356 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:03.535522 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.535531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.536512 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.536517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.537551 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.537566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.538618 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.538627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.538851 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.540072 (+ 1221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:03.540091 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:03.542952 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:03.546783 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.546788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.546791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.547284 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.547734 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.547739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.547745 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.547944 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.548660 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:03.548664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:03.548857 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 15:50:03.549001 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 15:50:03.549320 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 15:50:03.549449 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.516799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.13.5.126:43855)
0928 15:50:03.517465 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.517470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.517492 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:50:03.517540 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.517546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.517624 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.517726 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.519049 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.520213 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.521456 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.521978 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.522113 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.542471 (+ 20358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.542476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.542479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.542517 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.545957 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.546662 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.546668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.546672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.546922 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.547401 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.547406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.547408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.547816 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.548168 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.548710 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.548991 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"spinlock_wait_cycles":34048,"thread_start_us":275,"threads_started":1}
I20260928 15:50:03.550294 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.534479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.13.5.124:43575)
0928 15:50:03.534635 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.534639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.534655 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:50:03.534727 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.534730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.534784 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.534864 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.535651 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.536375 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.537764 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.538396 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.538635 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.543163 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.543169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.543174 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.543211 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.546664 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.547422 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.547432 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.547435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.547587 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.548100 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.548106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.548110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.548473 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.549583 (+ 1110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.549689 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.549951 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:50:03.557093 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616: Bootstrap starting.
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610603, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:50:03.561218 3433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:03.562682 3433 log.cc:824] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:03.565800 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.550890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.13.5.124:43575)
0928 15:50:03.551082 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.551087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.551109 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:50:03.551155 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.551161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.551227 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.551322 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.552284 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.553087 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.554410 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.555081 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.555221 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.559053 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.559060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.559065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.559103 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.562571 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.563426 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.563433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.563437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.563586 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.564028 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.564038 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.564040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.564522 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.564689 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.565178 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.565416 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:50:03.565943 3433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616: No bootstrap required, opened a new log
I20260928 15:50:03.566708 3588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.550081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55658)
0928 15:50:03.550305 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.550326 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:03.550437 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:03.551343 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:03.551348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:03.551923 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:03.552128 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.552157 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.553227 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.553232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.554263 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.554276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.555264 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.555272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.555506 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.556216 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:03.556242 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:03.558819 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:03.562734 (+ 3915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.562743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.562749 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.563289 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.563719 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.563723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.563727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.563924 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.564740 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:03.564744 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:03.564901 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 15:50:03.566191 (+ 1290us) client_negotiation.cc:240] Negotiation successful
0928 15:50:03.566434 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 15:50:03.568988 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.551950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.13.5.125:40765)
0928 15:50:03.552122 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.552130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.552147 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:50:03.552202 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.552205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.552293 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.552397 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.554102 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.554956 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.556760 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.557367 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.557506 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.562257 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.562262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.562266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.562299 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.565764 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.566758 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.566764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.566768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.566937 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.567414 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.567420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.567423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.567792 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.568145 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.568422 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.568655 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:50:03.572840 3583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8: Bootstrap starting.
I20260928 15:50:03.573843 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:50:03.576527 3408 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "380e36f52a4a48a2863de8a3c1f02616"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54240
I20260928 15:50:03.577113 3583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:03.577541 3433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } }
I20260928 15:50:03.577970 3433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380e36f52a4a48a2863de8a3c1f02616, State: Initialized, Role: FOLLOWER
I20260928 15:50:03.578550 3583 log.cc:824] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:03.578550 3433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } }
I20260928 15:50:03.579609 3591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.580119 3591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:03.580878 3433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:03.581039 3408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:50:03.581261 3408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:03.582695 3583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8: No bootstrap required, opened a new log
I20260928 15:50:03.584057 3408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } }
I20260928 15:50:03.586197 3408 leader_election.cc:288] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [CANDIDATE]: Term 1 election: Requested vote from peers bdd5e7ab2de345bb87953049895c4793 (127.13.5.125:40765), f6d093fe6ded4949a8e3f46ff9cf71d8 (127.13.5.124:43575)
I20260928 15:50:03.599107 3583 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } }
I20260928 15:50:03.599761 3583 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d093fe6ded4949a8e3f46ff9cf71d8, State: Initialized, Role: FOLLOWER
I20260928 15:50:03.600596 3583 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } }
I20260928 15:50:03.602826 3599 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.604159 3557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "380e36f52a4a48a2863de8a3c1f02616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8"
I20260928 15:50:03.604532 3599 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:03.604875 3557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:03.609982 3557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 380e36f52a4a48a2863de8a3c1f02616 in term 1.
I20260928 15:50:03.604133 3583 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:03.611765 3382 leader_election.cc:302] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380e36f52a4a48a2863de8a3c1f02616, f6d093fe6ded4949a8e3f46ff9cf71d8; no voters:
I20260928 15:50:03.612398 3591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [term 1 FOLLOWER]: Leader election won for term 1
W20260928 15:50:03.631649 3387 tablet.cc:2369] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:03.632812 3591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [term 1 LEADER]: Becoming Leader. State: Replica: 380e36f52a4a48a2863de8a3c1f02616, State: Running, Role: LEADER
I20260928 15:50:03.633617 3591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } }
I20260928 15:50:03.649127 3605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380e36f52a4a48a2863de8a3c1f02616. Latest consensus state: current_term: 1 leader_uuid: "380e36f52a4a48a2863de8a3c1f02616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.649685 3605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:03.659972 3619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:03.661865 3621 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:50:03.670403 3507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } attempt: 5
I20260928 15:50:03.674943 3619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:03.667708 3366 master_runner.cc:186] Error getting master registration for 127.13.5.125:40765: OK, instance_id { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" instance_seqno: 1790610602939572 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:50:03.683774 3366 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:50:03.695694 3557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [term 1 FOLLOWER]: Refusing update from remote peer 380e36f52a4a48a2863de8a3c1f02616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:03.696825 3591 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 28 15:50:03 dist-test-slave-j968 krb5kdc[3352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610602, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
I20260928 15:50:03.709190 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.613380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.13.5.125:40765)
0928 15:50:03.613982 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.613992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.642427 (+ 28435us) server_negotiation.cc:407] Connection header received
0928 15:50:03.644090 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.644100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.644210 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.644300 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.655668 (+ 11368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.657166 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.665520 (+ 8354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.666292 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.666412 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.693323 (+ 26911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.693331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.693335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.693383 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.701791 (+ 8408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.705716 (+ 3925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.705724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.705728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.705906 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.706434 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.706441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.706446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.706868 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.707152 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.708479 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.708771 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260928 15:50:03.709901 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.613706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.13.5.126:43855)
0928 15:50:03.614575 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.614585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.642280 (+ 27695us) server_negotiation.cc:407] Connection header received
0928 15:50:03.643443 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.643453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.643573 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.643670 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.653747 (+ 10077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.655324 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.666726 (+ 11402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.667592 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.667716 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.696095 (+ 28379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.696102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.696107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.696152 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.701131 (+ 4979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.705734 (+ 4603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.705743 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.705749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.706057 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.706631 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.706638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.706644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.707094 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.707401 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.709019 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.709406 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538}
W20260928 15:50:03.717482 3536 tablet.cc:2369] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:03.719038 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.614162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.13.5.124:43575)
0928 15:50:03.614791 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.614800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.669343 (+ 54543us) server_negotiation.cc:407] Connection header received
0928 15:50:03.670692 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.670700 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.670798 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.670875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.675955 (+ 5080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.677282 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.699128 (+ 21846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.699814 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.699992 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.703829 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.703838 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.703846 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.703916 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.708090 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.712701 (+ 4611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.712708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.712713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.712888 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.714562 (+ 1674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.714568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.714572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.715053 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.715330 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.718119 (+ 2789us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.718412 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462}
I20260928 15:50:03.720296 3599 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380e36f52a4a48a2863de8a3c1f02616. Latest consensus state: current_term: 1 leader_uuid: "380e36f52a4a48a2863de8a3c1f02616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.720661 3599 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:03.721959 3628 mvcc.cc:205] Tried to move back new op lower bound from 7334341032718196736 to 7334341032491540480. Current Snapshot: MvccSnapshot[applied={T|T < 7334341032718196736}]
I20260928 15:50:03.728693 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.670991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:43575 (local address 127.0.0.1:55682)
0928 15:50:03.671349 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:03.671368 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:03.671471 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:03.688302 (+ 16831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:03.688307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:03.688865 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:03.689146 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.689164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.690962 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.690967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.692399 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:03.692419 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.693942 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.693948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:03.694195 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.696886 (+ 2691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:03.696909 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:03.705692 (+ 8783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:03.718340 (+ 12648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.718348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.718352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.718833 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.721042 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:03.721051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:03.721057 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:03.721349 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:03.722952 (+ 1603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:03.722963 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:03.723345 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 15:50:03.723638 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 15:50:03.728075 (+ 4437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260928 15:50:03.728633 3605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "380e36f52a4a48a2863de8a3c1f02616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.729101 3605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:03.730118 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:03.671676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.13.5.124:43575)
0928 15:50:03.687683 (+ 16007us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.687692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.687727 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:50:03.687807 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.687813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.687949 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.689334 (+ 1385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.689614 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.690781 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.692621 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:03.693341 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:03.713059 (+ 19718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:03.713326 (+ 267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:03.713332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:03.713349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:03.713384 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:03.717999 (+ 4615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.718969 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.718974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.718977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.720834 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:03.721662 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:03.721669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:03.721676 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:03.722668 (+ 992us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:03.729054 (+ 6386us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:03.729260 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 15:50:03.729569 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15828,"thread_start_us":16396,"threads_started":1}
I20260928 15:50:03.730870 3599 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "380e36f52a4a48a2863de8a3c1f02616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.731181 3599 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:03.738252 3507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793: Bootstrap starting.
I20260928 15:50:03.738085 3629 mvcc.cc:205] Tried to move back new op lower bound from 7334341032718196736 to 7334341032491540480. Current Snapshot: MvccSnapshot[applied={T|T < 7334341032718196736}]
I20260928 15:50:03.743575 3507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:03.744436 3619 catalog_manager.cc:1347] Generated new cluster ID: 6dda8da4a2ae4abe901bf6b0209a7d97
I20260928 15:50:03.744557 3619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:03.745332 3507 log.cc:824] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:03.789227 3619 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:03.776795 3515 master_runner.cc:186] Error getting master registration for 127.13.5.125:40765: OK, instance_id { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" instance_seqno: 1790610602939572 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:50:03.790122 3515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:50:03.791657 3619 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:03.794368 3507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793: No bootstrap required, opened a new log
I20260928 15:50:03.796829 3631 catalog_manager.cc:1259] Loaded cluster ID: 6dda8da4a2ae4abe901bf6b0209a7d97
I20260928 15:50:03.796934 3631 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8: loading cluster ID for follower catalog manager: success
I20260928 15:50:03.801466 3631 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8: acquiring CA information for follower catalog manager: success
W20260928 15:50:03.803105 3631 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260928 15:50:03.820195 3619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616: Generated new TSK 0
I20260928 15:50:03.818817 3507 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } }
I20260928 15:50:03.820901 3507 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd5e7ab2de345bb87953049895c4793, State: Initialized, Role: FOLLOWER
I20260928 15:50:03.821758 3619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:50:03.822039 3507 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } }
I20260928 15:50:03.824492 3636 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.825119 3636 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:03.826648 3507 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:03.826725 3482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "380e36f52a4a48a2863de8a3c1f02616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bdd5e7ab2de345bb87953049895c4793"
I20260928 15:50:03.827251 3482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:03.833834 3482 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 380e36f52a4a48a2863de8a3c1f02616 in term 1.
I20260928 15:50:03.834863 3481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [term 1 FOLLOWER]: Refusing update from remote peer 380e36f52a4a48a2863de8a3c1f02616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:03.840502 3591 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:03.875898 3636 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380e36f52a4a48a2863de8a3c1f02616. Latest consensus state: current_term: 1 leader_uuid: "380e36f52a4a48a2863de8a3c1f02616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.880091 3636 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:03.879976 3643 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "380e36f52a4a48a2863de8a3c1f02616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:03.880591 3643 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:03.931763 3654 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 15:50:03.984221 3589 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:03.961602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.13.5.126:43855)
0928 15:50:03.961812 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.961818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.982012 (+ 20194us) server_negotiation.cc:407] Connection header received
0928 15:50:03.982371 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.982379 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.982469 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.982548 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.983849 (+ 1301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54266: BlockingRecv error: recv got EOF from 127.0.0.1:54266 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 15:50:03.988288 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:03.980619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.13.5.125:40765)
0928 15:50:03.980840 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.980847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.985771 (+ 4924us) server_negotiation.cc:407] Connection header received
0928 15:50:03.986500 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.986505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.986597 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.986667 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.988037 (+ 1370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60646: BlockingRecv error: recv got EOF from 127.0.0.1:60646 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 15:50:03.988758 3627 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:03.968170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.13.5.124:43575)
0928 15:50:03.968368 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.968374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.985067 (+ 16693us) server_negotiation.cc:407] Connection header received
0928 15:50:03.985895 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.985902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.986044 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.986162 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.988513 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55692: BlockingRecv error: recv got EOF from 127.0.0.1:55692 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 15:50:03.997819 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:03.995170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.13.5.125:40765)
0928 15:50:03.995346 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:03.995352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:03.996169 (+ 817us) server_negotiation.cc:407] Connection header received
0928 15:50:03.996531 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:03.996538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:03.996683 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:03.996798 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:03.997536 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60650: BlockingRecv error: recv got EOF from 127.0.0.1:60650 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 15:50:04.018844 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.010186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.13.5.125:40765)
0928 15:50:04.010381 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.010387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.016880 (+ 6493us) server_negotiation.cc:407] Connection header received
0928 15:50:04.017166 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.017173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.017275 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.017354 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.018607 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60662: BlockingRecv error: recv got EOF from 127.0.0.1:60662 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 15:50:04.026880 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.024670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.13.5.125:40765)
0928 15:50:04.024842 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.024848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.025355 (+ 507us) server_negotiation.cc:407] Connection header received
0928 15:50:04.025566 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.025572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.025681 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.025794 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.026600 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60670: BlockingRecv error: recv got EOF from 127.0.0.1:60670 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 15:50:04.040220 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.037750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.13.5.125:40765)
0928 15:50:04.037941 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.037947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.038583 (+ 636us) server_negotiation.cc:407] Connection header received
0928 15:50:04.038895 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.038901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.039012 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.039125 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.039990 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60684: BlockingRecv error: recv got EOF from 127.0.0.1:60684 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 15:50:04.054086 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.051906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.13.5.125:40765)
0928 15:50:04.052101 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.052107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.052615 (+ 508us) server_negotiation.cc:407] Connection header received
0928 15:50:04.052816 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.052821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.052915 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.053023 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.053847 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60688: BlockingRecv error: recv got EOF from 127.0.0.1:60688 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 15:50:04.080081 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.077663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.13.5.125:40765)
0928 15:50:04.077861 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.077868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.078453 (+ 585us) server_negotiation.cc:407] Connection header received
0928 15:50:04.078650 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.078657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.078759 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.078877 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.079798 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60700: BlockingRecv error: recv got EOF from 127.0.0.1:60700 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 15:50:04.118338 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.116006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.13.5.125:40765)
0928 15:50:04.116199 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.116206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.116753 (+ 547us) server_negotiation.cc:407] Connection header received
0928 15:50:04.116931 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.116937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.117065 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.117194 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.118057 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60702: BlockingRecv error: recv got EOF from 127.0.0.1:60702 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 15:50:04.194892 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.192434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.13.5.125:40765)
0928 15:50:04.192654 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.192660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.193246 (+ 586us) server_negotiation.cc:407] Connection header received
0928 15:50:04.193462 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.193469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.193588 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.193709 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.194612 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60718: BlockingRecv error: recv got EOF from 127.0.0.1:60718 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 15:50:04.329047 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.326745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.13.5.125:40765)
0928 15:50:04.326952 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.326961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.327329 (+ 368us) server_negotiation.cc:407] Connection header received
0928 15:50:04.327525 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.327529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.327643 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.327744 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.328824 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60722: BlockingRecv error: recv got EOF from 127.0.0.1:60722 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:50:04.381309 3591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "380e36f52a4a48a2863de8a3c1f02616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380e36f52a4a48a2863de8a3c1f02616" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 43855 } } peers { permanent_uuid: "bdd5e7ab2de345bb87953049895c4793" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 40765 } } peers { permanent_uuid: "f6d093fe6ded4949a8e3f46ff9cf71d8" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 43575 } } }
I20260928 15:50:04.381690 3591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380e36f52a4a48a2863de8a3c1f02616 [sys.catalog]: This master's current role is: LEADER
W20260928 15:50:04.591643 3513 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:04.589632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.13.5.125:40765)
0928 15:50:04.589924 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:04.589930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:04.590154 (+ 224us) server_negotiation.cc:407] Connection header received
0928 15:50:04.590417 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:04.590422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:04.590498 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:04.590613 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:04.591394 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60732: BlockingRecv error: recv got EOF from 127.0.0.1:60732 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 15:50:04.805932 3631 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6d093fe6ded4949a8e3f46ff9cf71d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:50:04.936255 3654 catalog_manager.cc:1259] Loaded cluster ID: 6dda8da4a2ae4abe901bf6b0209a7d97
I20260928 15:50:04.936379 3654 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793: loading cluster ID for follower catalog manager: success
I20260928 15:50:04.939363 3654 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793: acquiring CA information for follower catalog manager: success
I20260928 15:50:04.941322 3654 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bdd5e7ab2de345bb87953049895c4793: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:50:05.119059 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:05.109679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.13.5.125:40765)
0928 15:50:05.111185 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:05.111192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:05.111453 (+ 261us) server_negotiation.cc:407] Connection header received
0928 15:50:05.111755 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:05.111759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:05.111986 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:05.112133 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:05.112664 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.113575 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.116028 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.116762 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.116876 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:05.117633 (+ 757us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:05.118223 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:05.118338 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:05.118503 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 15:50:05.118612 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":1045,"threads_started":1}
I20260928 15:50:05.133744 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:05.125972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.13.5.124:43575)
0928 15:50:05.128296 (+ 2324us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:05.128301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:05.128318 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:50:05.128428 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:05.128432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:05.128609 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:05.128795 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:05.129232 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.130010 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.131402 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.131915 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.132050 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:05.132551 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:05.133112 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:05.133295 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:05.133346 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 15:50:05.133433 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2210,"thread_start_us":2061,"threads_started":1}
I20260928 15:50:05.151799 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:05.140881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.13.5.125:40765)
0928 15:50:05.141089 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:05.141096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:05.144517 (+ 3421us) server_negotiation.cc:407] Connection header received
0928 15:50:05.144887 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:05.144896 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:05.145023 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:05.145103 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:05.146047 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.147646 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.149455 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.150108 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.150209 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:05.150890 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:05.151040 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:05.151109 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:05.151371 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 15:50:05.151466 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:50:05.155934 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:05.141929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.13.5.126:43855)
0928 15:50:05.148338 (+ 6409us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:05.148347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:05.148374 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:50:05.148516 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:05.148524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:05.148805 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:05.149039 (+ 234us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:05.149681 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.150739 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.152776 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.153413 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.154032 (+ 619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:05.154286 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:05.154707 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:05.155427 (+ 720us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:05.155483 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 15:50:05.155573 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6230,"thread_start_us":6112,"threads_started":1}
I20260928 15:50:05.159181 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:05.141893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.13.5.124:43575)
0928 15:50:05.142120 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:05.142126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:05.152634 (+ 10508us) server_negotiation.cc:407] Connection header received
0928 15:50:05.152907 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:05.152913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:05.153004 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:05.153066 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:05.153690 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.154714 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.156553 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:05.157144 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:05.157242 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:05.158234 (+ 992us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:05.158385 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:05.158461 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:05.158787 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 15:50:05.158883 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:50:05.161154 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3366
I20260928 15:50:05.162452 3428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:05.276135 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3366
I20260928 15:50:05.299340 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3440
I20260928 15:50:05.300700 3502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:05.420276 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3440
I20260928 15:50:05.442018 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3515
I20260928 15:50:05.443183 3578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:05.557040 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3515
2026-09-28T15:50:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5047 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:05 dist-test-slave-j968 krb5kdc[3741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:05 dist-test-slave-j968 krb5kdc[3741](info): set up 2 sockets
Sep 28 15:50:05 dist-test-slave-j968 krb5kdc[3741](info): commencing operation
krb5kdc: starting...
W20260928 15:50:06.695039 13333 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/test-user.keytab.
Sep 28 15:50:06 dist-test-slave-j968 krb5kdc[3741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.5.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
Entry for principal kudu/127.13.5.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab.
I20260928 15:50:06.980479 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:39485
--webserver_interface=127.13.5.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:39485,127.13.5.125:41727,127.13.5.124:33271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:07.339735 3754 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:50:07.340193 3754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:07.340277 3754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:07.340366 3754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:07.347820 3754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:07.347949 3754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:07.348009 3754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:07.348054 3754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:07.348129 3754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:07.358419 3754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:39485,127.13.5.125:41727,127.13.5.124:33271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.126.keytab
--principal=kudu/127.13.5.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.126:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.5.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:07.359764 3754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:07.361155 3754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:07.373530 3760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:07.373530 3762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:07.373554 3759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:07.374212 3754 server_base.cc:1034] running on GCE node
I20260928 15:50:07.375082 3754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:07.376688 3754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:07.377964 3754 hybrid_clock.cc:648] HybridClock initialized: now 1790610607377895 us; error 36 us; skew 500 ppm
I20260928 15:50:07.378331 3754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:07 dist-test-slave-j968 krb5kdc[3741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:07.381852 3754 init.cc:377] Logged in from keytab as kudu/127.13.5.126@KRBTEST.COM (short username kudu)
I20260928 15:50:07.383447 3754 webserver.cc:460] Webserver started at http://127.13.5.126:36351/ using document root <none> and password file <none>
I20260928 15:50:07.384279 3754 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:07.384415 3754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:07.384765 3754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:07.387826 3754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/data/instance:
uuid: "6cb1d8f40d30422f8406984fcdbbebfb"
format_stamp: "Formatted at 2026-09-28 15:50:07 on dist-test-slave-j968"
server_key: "938f1464fda0ae4f414fcbc1644c9651"
server_key_iv: "30b3f8712a6f5433c088431e7fe2196d"
server_key_version: "encryptionkey@0"
I20260928 15:50:07.388700 3754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/wal/instance:
uuid: "6cb1d8f40d30422f8406984fcdbbebfb"
format_stamp: "Formatted at 2026-09-28 15:50:07 on dist-test-slave-j968"
server_key: "938f1464fda0ae4f414fcbc1644c9651"
server_key_iv: "30b3f8712a6f5433c088431e7fe2196d"
server_key_version: "encryptionkey@0"
I20260928 15:50:07.394158 3754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:50:07.398023 3769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:07.399696 3754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:50:07.399994 3754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/wal
uuid: "6cb1d8f40d30422f8406984fcdbbebfb"
format_stamp: "Formatted at 2026-09-28 15:50:07 on dist-test-slave-j968"
server_key: "938f1464fda0ae4f414fcbc1644c9651"
server_key_iv: "30b3f8712a6f5433c088431e7fe2196d"
server_key_version: "encryptionkey@0"
I20260928 15:50:07.400245 3754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:07.413262 3754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:07.417667 3754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:07.417964 3754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:07.437884 3754 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.126:39485
I20260928 15:50:07.437927 3820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.126:39485 every 8 connection(s)
I20260928 15:50:07.439375 3754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:07.443672 3821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:07.448621 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3754
I20260928 15:50:07.448815 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-0/wal/instance
I20260928 15:50:07.449337 13333 external_mini_cluster.cc:1428] Setting key b9a53e4ed78a84656b65e1eb4e66bc7b
I20260928 15:50:07.451378 3821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:50:07 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.125@KRBTEST.COM" created.
I20260928 15:50:07.476478 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.455066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39485 (local address 127.0.0.1:39676)
0928 15:50:07.456053 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.456120 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:07.458376 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:07.459304 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:07.459312 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:07.460157 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:07.460522 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:07.460542 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:07.462276 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:07.462286 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:07.463438 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:07.463456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:07.464554 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:07.464562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:07.464759 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:07.465495 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:07.465525 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:07.467961 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:07.471650 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:07.471663 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:07.471678 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:07.472203 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:07.472733 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:07.472738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:07.472742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:07.472934 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:07.473921 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:07.473935 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:07.474205 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 15:50:07.474481 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 15:50:07.475269 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":242,"thread_start_us":417,"threads_started":1}
I20260928 15:50:07.476548 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.455069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.13.5.126:39485)
0928 15:50:07.456960 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:07.456969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:07.457027 (+ 58us) server_negotiation.cc:407] Connection header received
0928 15:50:07.458805 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:07.458811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:07.459065 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:07.459295 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:07.460755 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:07.462097 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:07.463600 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:07.464297 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:07.464529 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:07.468217 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:07.468235 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:07.468249 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:07.468289 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:07.471454 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:07.472355 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:07.472386 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:07.472390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:07.472597 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:07.473061 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:07.473068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:07.473071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:07.473775 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:07.474031 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:07.474640 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 15:50:07.475126 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":1643,"thread_start_us":1411,"threads_started":1}
I20260928 15:50:07.479746 3821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:07.481108 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.480669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41742)
0928 15:50:07.480852 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.480989 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 15:50:07.481551 3772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
I20260928 15:50:07.483325 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.482927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41758)
0928 15:50:07.483111 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.483184 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 15:50:07.483680 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:41727: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab.
Entry for principal kudu/127.13.5.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab.
I20260928 15:50:07.512416 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.125:41727
--webserver_interface=127.13.5.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab
--principal=kudu/127.13.5.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:39485,127.13.5.125:41727,127.13.5.124:33271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:50:07.533069 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } attempt: 1
I20260928 15:50:07.534300 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.533833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41760)
0928 15:50:07.534045 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.534157 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:50:07.535369 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.535007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41772)
0928 15:50:07.535155 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.535235 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:50:07.535567 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:41727: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
I20260928 15:50:07.603893 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } attempt: 2
I20260928 15:50:07.605369 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.604666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41778)
0928 15:50:07.605004 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.605145 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 15:50:07.606562 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.606133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41790)
0928 15:50:07.606293 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.606400 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:50:07.606796 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:41727: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
I20260928 15:50:07.698096 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } attempt: 3
I20260928 15:50:07.699599 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.699009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41802)
0928 15:50:07.699274 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.699400 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 15:50:07.700874 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.700385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41812)
0928 15:50:07.700565 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.700680 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:50:07.701217 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:41727: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
I20260928 15:50:07.844627 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } attempt: 4
I20260928 15:50:07.846041 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.845414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41814)
0928 15:50:07.845793 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.845891 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260928 15:50:07.847388 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:07.846902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41822)
0928 15:50:07.847092 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:07.847207 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 15:50:07.847633 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.125:41727: Network error: Client connection negotiation failed: client connection to 127.13.5.125:41727: connect: Connection refused (error 111)
W20260928 15:50:07.870162 3828 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:50:07.870573 3828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:07.870699 3828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:07.870831 3828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:07.878294 3828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:07.878427 3828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:07.878572 3828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:07.878659 3828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:07.878772 3828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:07.889191 3828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:39485,127.13.5.125:41727,127.13.5.124:33271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.125.keytab
--principal=kudu/127.13.5.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.125:41727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.5.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:07.890654 3828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:07.892225 3828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:07.904801 3836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:07.904829 3833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:07.904886 3834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:07.905630 3828 server_base.cc:1034] running on GCE node
I20260928 15:50:07.906509 3828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:07.908164 3828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:07.909430 3828 hybrid_clock.cc:648] HybridClock initialized: now 1790610607909359 us; error 45 us; skew 500 ppm
I20260928 15:50:07.909791 3828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:07 dist-test-slave-j968 krb5kdc[3741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:07.913271 3828 init.cc:377] Logged in from keytab as kudu/127.13.5.125@KRBTEST.COM (short username kudu)
I20260928 15:50:07.914829 3828 webserver.cc:460] Webserver started at http://127.13.5.125:41563/ using document root <none> and password file <none>
I20260928 15:50:07.915524 3828 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:07.915614 3828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:07.915956 3828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:07.919023 3828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/data/instance:
uuid: "1ab001d6f87d48fd9fe121a5cca33865"
format_stamp: "Formatted at 2026-09-28 15:50:07 on dist-test-slave-j968"
server_key: "e6cc1596d4a06ec639e2614496d742a0"
server_key_iv: "40ea8b9ba96d9732ddb249e54f6056ec"
server_key_version: "encryptionkey@0"
I20260928 15:50:07.919930 3828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/wal/instance:
uuid: "1ab001d6f87d48fd9fe121a5cca33865"
format_stamp: "Formatted at 2026-09-28 15:50:07 on dist-test-slave-j968"
server_key: "e6cc1596d4a06ec639e2614496d742a0"
server_key_iv: "40ea8b9ba96d9732ddb249e54f6056ec"
server_key_version: "encryptionkey@0"
I20260928 15:50:07.925688 3828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:50:07.929605 3843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:07.931176 3828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:50:07.931399 3828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/wal
uuid: "1ab001d6f87d48fd9fe121a5cca33865"
format_stamp: "Formatted at 2026-09-28 15:50:07 on dist-test-slave-j968"
server_key: "e6cc1596d4a06ec639e2614496d742a0"
server_key_iv: "40ea8b9ba96d9732ddb249e54f6056ec"
server_key_version: "encryptionkey@0"
I20260928 15:50:07.931703 3828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:07.957216 3828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:07.961962 3828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:07.962252 3828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:07.982537 3828 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.125:41727
I20260928 15:50:07.982645 3896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.125:41727 every 8 connection(s)
I20260928 15:50:07.984071 3828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/data/info.pb
I20260928 15:50:07.988219 3897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:07.990604 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3828
I20260928 15:50:07.990831 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-1/wal/instance
I20260928 15:50:07.991495 13333 external_mini_cluster.cc:1428] Setting key cce63fbcfe8a44ec13c84b6ebcfd688a
I20260928 15:50:07.998842 3897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.5.124@KRBTEST.COM; defaulting to no policy
I20260928 15:50:08.021893 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.002665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.13.5.126:39485)
0928 15:50:08.003744 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.003755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.004583 (+ 828us) server_negotiation.cc:407] Connection header received
0928 15:50:08.006644 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.006649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.006744 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.006869 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.008582 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.009295 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.010844 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.011261 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.011409 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.015617 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.015622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.015625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.015672 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.018521 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.019343 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.019348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.019350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.019482 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.019991 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.019994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.019997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.020295 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.020461 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.021254 (+ 793us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.021505 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":731,"threads_started":1}
I20260928 15:50:08.022959 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.003272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39485 (local address 127.0.0.1:39680)
0928 15:50:08.004386 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.004451 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:08.006345 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:08.007120 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:08.007129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:08.007654 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:08.008357 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.008375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.009466 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.009470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.010679 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.010692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.011505 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.011513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.011752 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.012622 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:08.012649 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:08.015362 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:08.018715 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.018724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.018742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.019198 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.019615 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.019621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.019625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.019831 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.020432 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:08.020441 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:08.020836 (+ 395us) client_negotiation.cc:769] Sending connection context
0928 15:50:08.021149 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 15:50:08.021704 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":386,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.5.124@KRBTEST.COM" created.
I20260928 15:50:08.024973 3897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:50:08.042510 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.025799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41836)
0928 15:50:08.026019 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.026039 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:08.026155 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:08.027146 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:08.027153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:08.027641 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:08.027846 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.027860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.029380 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.029387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.030881 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.030899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.031754 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.031762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.032008 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.032802 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:08.032830 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:08.035243 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:08.039842 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.039846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.039849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.040271 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.040709 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.040713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.040715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.040839 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.041747 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:08.041751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:08.041884 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 15:50:08.042016 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 15:50:08.042234 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:50:08.043037 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.026022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.13.5.125:41727)
0928 15:50:08.026709 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.026715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.026741 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:50:08.026785 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.026791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.026972 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.027093 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.028050 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.029232 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.031021 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.031552 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.031686 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.035497 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.035515 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.035538 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.035596 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.039689 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.040406 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.040418 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.040424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.040588 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.040957 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.040965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.040970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.041518 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.041794 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.042205 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.042496 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
I20260928 15:50:08.045914 3897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:08.047179 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.046725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60088)
0928 15:50:08.046907 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.047037 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 15:50:08.047488 3847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
I20260928 15:50:08.049232 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.048891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60090)
0928 15:50:08.049043 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.049126 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:50:08.049445 3897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.5.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab.
Entry for principal kudu/127.13.5.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab.
I20260928 15:50:08.061396 13333 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
/tmp/dist-test-task6Pa7uo/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.124:33271
--webserver_interface=127.13.5.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab
--principal=kudu/127.13.5.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.5.84:38383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.5.126:39485,127.13.5.125:41727,127.13.5.124:33271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:50:08.098768 3897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 1
I20260928 15:50:08.100081 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.099496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60104)
0928 15:50:08.099730 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.099880 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 15:50:08.101197 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.100779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60110)
0928 15:50:08.100963 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.101058 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 15:50:08.101441 3897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
I20260928 15:50:08.146924 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } attempt: 5
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:50:08.160555 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.147849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.13.5.125:41727)
0928 15:50:08.148100 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.148106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.148130 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:50:08.148208 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.148214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.148310 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.148419 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.149235 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.149947 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.151123 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.151603 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.151737 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.155240 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.155245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.155248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.155283 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.157911 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.158581 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.158590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.158595 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.158720 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.159105 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.159111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.159114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.159412 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.159578 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.159982 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.160212 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 15:50:08.160555 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.147722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:41848)
0928 15:50:08.147941 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.147964 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:08.148092 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:08.148465 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:08.148470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:08.148898 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:08.149078 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.149087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.150085 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.150093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.150972 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.150985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.151763 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.151769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.152030 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.152716 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:08.152734 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:08.155031 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:08.158067 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.158073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.158077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.158441 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.158845 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.158850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.158853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.158989 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.159536 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:08.159541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:08.159710 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 15:50:08.159856 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:50:08.160194 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 15:50:08.161708 3821 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:08.162819 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.162337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60112)
0928 15:50:08.162539 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.162633 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 15:50:08.163797 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.163416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60116)
0928 15:50:08.163565 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.163664 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:50:08.164022 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
I20260928 15:50:08.169667 3897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 2
I20260928 15:50:08.170718 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.170255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60122)
0928 15:50:08.170450 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.170548 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:50:08.171783 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.171407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60136)
0928 15:50:08.171558 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.171640 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:50:08.172020 3897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
I20260928 15:50:08.215265 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 1
I20260928 15:50:08.216598 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.216033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60150)
0928 15:50:08.216271 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.216383 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 15:50:08.217672 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.217301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60166)
0928 15:50:08.217457 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.217541 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:50:08.217936 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
I20260928 15:50:08.263293 3897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 3
I20260928 15:50:08.264765 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.264097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60170)
0928 15:50:08.264384 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.264522 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 15:50:08.266088 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.265493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60178)
0928 15:50:08.265787 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.265914 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260928 15:50:08.266330 3897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
I20260928 15:50:08.286178 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 2
I20260928 15:50:08.287421 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.286933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60180)
0928 15:50:08.287163 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.287256 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:50:08.288669 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.288203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:60190)
0928 15:50:08.288387 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.288528 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 15:50:08.289002 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
I20260928 15:50:08.395287 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 3
I20260928 15:50:08.396539 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.396033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:58020)
0928 15:50:08.396261 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.396367 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 15:50:08.397693 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.397277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:58030)
0928 15:50:08.397500 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.397578 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 15:50:08.397970 3821 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
I20260928 15:50:08.409579 3897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 4
I20260928 15:50:08.410755 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.410213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:58044)
0928 15:50:08.410463 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.410566 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:50:08.411752 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.411390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:58050)
0928 15:50:08.411544 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.411627 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:50:08.411990 3897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.5.124:33271: Network error: Client connection negotiation failed: client connection to 127.13.5.124:33271: connect: Connection refused (error 111)
W20260928 15:50:08.412007 3905 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:50:08.412433 3905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:08.412535 3905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:08.412668 3905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:08.420146 3905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:08.420235 3905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:08.420327 3905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:08.420540 3905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:08.420662 3905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:08.431842 3905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.5.84:38383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.5.126:39485,127.13.5.125:41727,127.13.5.124:33271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/krb5kdc/kudu_127.13.5.124.keytab
--principal=kudu/127.13.5.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.5.124:33271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.5.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:08.433327 3905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:08.434715 3905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:08.447283 3910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:08.447328 3911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:08.447968 3913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:08.448403 3905 server_base.cc:1034] running on GCE node
I20260928 15:50:08.449189 3905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:08.450821 3905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:08.452059 3905 hybrid_clock.cc:648] HybridClock initialized: now 1790610608451995 us; error 46 us; skew 500 ppm
I20260928 15:50:08.452385 3905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610608, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:08.455932 3905 init.cc:377] Logged in from keytab as kudu/127.13.5.124@KRBTEST.COM (short username kudu)
I20260928 15:50:08.457571 3905 webserver.cc:460] Webserver started at http://127.13.5.124:43991/ using document root <none> and password file <none>
I20260928 15:50:08.458307 3905 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:08.458395 3905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:08.458668 3905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:08.461723 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/data/instance:
uuid: "85323e5f198d4a24b85b90d15a0c928b"
format_stamp: "Formatted at 2026-09-28 15:50:08 on dist-test-slave-j968"
server_key: "f58ef176a3b1eb4204108ecc849d0db0"
server_key_iv: "6aca2e8756a6d37eb3f5f7c4e104a857"
server_key_version: "encryptionkey@0"
I20260928 15:50:08.462532 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/wal/instance:
uuid: "85323e5f198d4a24b85b90d15a0c928b"
format_stamp: "Formatted at 2026-09-28 15:50:08 on dist-test-slave-j968"
server_key: "f58ef176a3b1eb4204108ecc849d0db0"
server_key_iv: "6aca2e8756a6d37eb3f5f7c4e104a857"
server_key_version: "encryptionkey@0"
I20260928 15:50:08.468401 3905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 15:50:08.472132 3920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:08.473464 3905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:50:08.473649 3905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/data,/tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/wal
uuid: "85323e5f198d4a24b85b90d15a0c928b"
format_stamp: "Formatted at 2026-09-28 15:50:08 on dist-test-slave-j968"
server_key: "f58ef176a3b1eb4204108ecc849d0db0"
server_key_iv: "6aca2e8756a6d37eb3f5f7c4e104a857"
server_key_version: "encryptionkey@0"
I20260928 15:50:08.473879 3905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:08.496717 3905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:08.501151 3905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:08.501435 3905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:08.520380 3905 rpc_server.cc:307] RPC server started. Bound to: 127.13.5.124:33271
I20260928 15:50:08.520401 3971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.5.124:33271 every 8 connection(s)
I20260928 15:50:08.521961 3905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/data/info.pb
I20260928 15:50:08.525996 3972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:08.527650 13333 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu as pid 3905
I20260928 15:50:08.527837 13333 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Pa7uo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610238672534-13333-0/minicluster-data/master-2/wal/instance
I20260928 15:50:08.528647 13333 external_mini_cluster.cc:1428] Setting key dfa4db5c899bc1682e3aa4e6aeb7279a
I20260928 15:50:08.533676 3972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610608, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:50:08.551398 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.536259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.13.5.126:39485)
0928 15:50:08.536803 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.536809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.537293 (+ 484us) server_negotiation.cc:407] Connection header received
0928 15:50:08.537474 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.537478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.537543 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.537664 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.538772 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.539463 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.541039 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.541450 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.541586 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.545106 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.545110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.545114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.545158 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.548554 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.549230 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.549236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.549240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.549383 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.549766 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.549772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.549775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.550065 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.550222 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.550669 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.550928 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
I20260928 15:50:08.552135 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.530673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.13.5.124:33271)
0928 15:50:08.531487 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.531503 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.531547 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:50:08.533255 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.533277 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.533659 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.534068 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.535527 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.537448 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.538729 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.539373 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.539569 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.543736 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.543744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.543755 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.543788 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.547293 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.548428 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.548435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.548441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.548590 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.549015 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.549021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.549024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.549601 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.549921 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.550372 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.550854 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":608,"spinlock_wait_cycles":13696,"thread_start_us":354,"threads_started":1}
I20260928 15:50:08.552182 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.536274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.126:39485 (local address 127.0.0.1:41704)
0928 15:50:08.537148 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.537190 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:08.537334 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:08.537727 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:08.537734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:08.538375 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:08.538620 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.538629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.539617 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.539623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.540892 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.540904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.541668 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.541677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.541848 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.542429 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:08.542450 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:08.544835 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:08.548683 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.548701 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.548708 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.549110 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.549502 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.549507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.549510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.549647 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.550200 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:08.550208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:08.550390 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 15:50:08.550575 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 15:50:08.551018 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":119,"thread_start_us":504,"threads_started":1}
I20260928 15:50:08.553797 3972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610608, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.126@KRBTEST.COM
I20260928 15:50:08.568578 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.554500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.125:41727 (local address 127.0.0.1:58160)
0928 15:50:08.554672 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.554690 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:08.554800 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:08.555125 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:08.555130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:08.555634 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:08.555831 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.555844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.557215 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.557218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.558443 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.558458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.559316 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.559321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.559472 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.560095 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:08.560121 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:08.562670 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:08.565950 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.565956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.565962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.566412 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.566823 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.566828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.566832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.566978 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.567510 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:08.567515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:08.567692 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 15:50:08.567916 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 15:50:08.568200 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 15:50:08.568614 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.554621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.13.5.125:41727)
0928 15:50:08.554829 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.554835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.554853 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:50:08.554915 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.554921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.554993 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.555081 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.556024 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.557051 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.558588 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.559177 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.559303 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.562879 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.562889 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.562895 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.562942 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.565817 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.566554 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.566561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.566565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.566709 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.567097 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.567103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.567106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.567374 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.567540 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.568017 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.568223 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:50:08.569530 3972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:50:08.572291 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.555596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.13.5.126:39485)
0928 15:50:08.556066 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.556070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.556083 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:50:08.556129 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.556132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.556192 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.556262 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.557717 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.558709 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.560133 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.560803 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.560930 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.565500 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.565505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.565508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.565540 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.568695 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.569502 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.569507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.569511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.569722 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.570156 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.570161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.570165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.570485 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.570774 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.571445 (+ 671us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.571634 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338}
I20260928 15:50:08.575203 3821 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 4
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610608, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.124@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.126@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.5.125@KRBTEST.COM
I20260928 15:50:08.588670 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.570354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:58072)
0928 15:50:08.570541 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.570562 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:08.570651 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:08.570910 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:08.570915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:08.571385 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:08.571577 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.571589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.573061 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.573069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.574231 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.574247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.575190 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.575199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.575583 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.576983 (+ 1400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:08.577017 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:08.580153 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:08.585531 (+ 5378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.585536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.585540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.586134 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.586553 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.586558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.586561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.586742 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.587404 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:08.587409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:08.587593 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 15:50:08.587803 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 15:50:08.588179 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"spinlock_wait_cycles":2432}
I20260928 15:50:08.589170 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.570503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.13.5.124:33271)
0928 15:50:08.570671 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.570675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.570691 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:50:08.570730 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.570733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.570783 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.570855 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.571760 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.572899 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.574393 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.574974 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.575124 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.580370 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.580377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.580395 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.580438 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.584290 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.586271 (+ 1981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.586275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.586280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.586421 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.586880 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.586886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.586898 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.587247 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.587407 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.587970 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.588763 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":78}
I20260928 15:50:08.592088 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.575818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:58080)
0928 15:50:08.576069 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.576091 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:08.576203 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:08.577323 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:08.577328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:08.577795 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:08.577997 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.578010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.579916 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.579922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.581206 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.581226 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.581978 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.581984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.582192 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.582781 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:08.582798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:08.585235 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:08.589310 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.589319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.589323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.589824 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.590325 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.590331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.590335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.590479 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.591151 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:08.591157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:08.591323 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 15:50:08.591473 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 15:50:08.591720 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 15:50:08.593040 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.574250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.13.5.125:41727)
0928 15:50:08.574426 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.574432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.574453 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:50:08.574618 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.574622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.574684 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.574773 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.576446 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.577821 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.579508 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.580025 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.580148 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.586036 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.586046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.586052 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.586096 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.590031 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.590853 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.590860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.590864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.591024 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.591518 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.591523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.591528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.591891 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.592170 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.592441 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.592694 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:50:08.593108 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.576424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.13.5.124:33271)
0928 15:50:08.577023 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.577027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.577044 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:50:08.577079 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.577082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.577130 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.577251 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.578185 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.579738 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.581367 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.581811 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.581966 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.585525 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.585531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.585534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.585600 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.589147 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.590007 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.590012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.590014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.590166 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.590632 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.590636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.590639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.590994 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.591138 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.592404 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.592630 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"spinlock_wait_cycles":49920,"thread_start_us":285,"threads_started":1}
I20260928 15:50:08.595690 3972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: Bootstrap starting.
I20260928 15:50:08.598258 13333 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:50:08.599819 3972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:08.601541 3972 log.cc:824] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:08.605335 3972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: No bootstrap required, opened a new log
I20260928 15:50:08.601573 3796 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6cb1d8f40d30422f8406984fcdbbebfb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41706
I20260928 15:50:08.605949 3821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb: Bootstrap starting.
I20260928 15:50:08.609907 3821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:08.611212 3821 log.cc:824] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:08.614660 3821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb: No bootstrap required, opened a new log
I20260928 15:50:08.615839 3972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } }
I20260928 15:50:08.616351 3972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85323e5f198d4a24b85b90d15a0c928b, State: Initialized, Role: FOLLOWER
I20260928 15:50:08.616882 3972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } }
I20260928 15:50:08.617941 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.618387 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:08.619191 3972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:08.623144 3821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } }
I20260928 15:50:08.623688 3821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb1d8f40d30422f8406984fcdbbebfb, State: Initialized, Role: FOLLOWER
I20260928 15:50:08.624365 3821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } }
I20260928 15:50:08.625478 3986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.625897 3986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:08.627236 3821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:08.627285 3796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:50:08.627498 3796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:08.631374 3796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } }
I20260928 15:50:08.633930 3796 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [CANDIDATE]: Term 1 election: Requested vote from peers 1ab001d6f87d48fd9fe121a5cca33865 (127.13.5.125:41727), 85323e5f198d4a24b85b90d15a0c928b (127.13.5.124:33271)
I20260928 15:50:08.634521 3947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85323e5f198d4a24b85b90d15a0c928b"
I20260928 15:50:08.635102 3947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:50:08.639164 3995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:50:08.639317 3995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:50:08.639324 3947 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cb1d8f40d30422f8406984fcdbbebfb in term 1.
I20260928 15:50:08.640162 3773 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb1d8f40d30422f8406984fcdbbebfb, 85323e5f198d4a24b85b90d15a0c928b; no voters:
I20260928 15:50:08.640687 3986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:08.636451 3905 master_runner.cc:186] Error getting master registration for 127.13.5.126:39485: OK, instance_id { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" instance_seqno: 1790610607432165 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:50:08.650252 3905 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:50:08.652837 3986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [term 1 LEADER]: Becoming Leader. State: Replica: 6cb1d8f40d30422f8406984fcdbbebfb, State: Running, Role: LEADER
I20260928 15:50:08.653616 3986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } }
I20260928 15:50:08.659114 3998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb1d8f40d30422f8406984fcdbbebfb. Latest consensus state: current_term: 1 leader_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.660208 3998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:08.664086 4008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:08.670552 4008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 15:50:08.671159 3775 tablet.cc:2369] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:08.711206 3897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } attempt: 5
Sep 28 15:50:08 dist-test-slave-j968 krb5kdc[3741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.5.125@KRBTEST.COM for kudu/127.13.5.124@KRBTEST.COM
I20260928 15:50:08.721499 3947 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [term 1 FOLLOWER]: Refusing update from remote peer 6cb1d8f40d30422f8406984fcdbbebfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:08.722647 3998 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:08.732630 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.673658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.13.5.124:33271)
0928 15:50:08.673900 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.673908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.691454 (+ 17546us) server_negotiation.cc:407] Connection header received
0928 15:50:08.691939 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.691945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.692054 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.692137 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.694234 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.695465 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.697857 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.698597 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.698706 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.708531 (+ 9825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.708539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.708545 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.708594 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.713764 (+ 5170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.722395 (+ 8631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.722401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.722405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.722594 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.723903 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.723909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.723913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.724577 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.731848 (+ 7271us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.732003 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.732240 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:50:08.732597 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.711958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.13.5.124:33271)
0928 15:50:08.712173 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.712179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.712386 (+ 207us) server_negotiation.cc:407] Connection header received
0928 15:50:08.712581 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.712587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.712683 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.712803 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.713772 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.714803 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.716668 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.717328 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.717457 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.722300 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.722307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.722311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.722349 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.726199 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.727497 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.727503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.727506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.727659 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.728566 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.728573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.728577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.729526 (+ 949us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.729819 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.730953 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.731552 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:50:08.735277 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.711898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.5.124:33271 (local address 127.0.0.1:58096)
0928 15:50:08.712300 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:08.712322 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:08.712452 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:08.712829 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:08.712835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:08.713389 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:08.713627 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.713641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.714963 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.714971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.716522 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:08.716538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.717504 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.717511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:08.717744 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.718414 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:08.718438 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:08.722062 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:08.726421 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.726428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.726433 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.727296 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.727859 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:08.727910 (+ 51us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:08.727914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:08.728247 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:08.729969 (+ 1722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:08.730000 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:08.730357 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 15:50:08.733335 (+ 2978us) client_negotiation.cc:240] Negotiation successful
0928 15:50:08.734145 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260928 15:50:08.736824 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.674054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.13.5.125:41727)
0928 15:50:08.674245 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.674250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.686760 (+ 12510us) server_negotiation.cc:407] Connection header received
0928 15:50:08.687227 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.687234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.687335 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.687412 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.700954 (+ 13542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.702343 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.704383 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.705044 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.705175 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.727264 (+ 22089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.727273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.727278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.727321 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.731574 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.733530 (+ 1956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.733538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.733544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.733734 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.734209 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.734215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.734218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.734629 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.734919 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.736163 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.736464 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:50:08.743186 3897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865: Bootstrap starting.
I20260928 15:50:08.714243 3754 master_runner.cc:186] Error getting master registration for 127.13.5.125:41727: OK, instance_id { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" instance_seqno: 1790610607976905 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:50:08.750900 3754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:50:08.751817 3897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865: Neither blocks nor log segments found. Creating new log.
W20260928 15:50:08.753531 3926 tablet.cc:2369] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:08.757992 3897 log.cc:824] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:08.759956 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:08.670072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.13.5.126:39485)
0928 15:50:08.670272 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:08.670279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:08.684790 (+ 14511us) server_negotiation.cc:407] Connection header received
0928 15:50:08.686926 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:08.686933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:08.687088 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:08.687176 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:08.694049 (+ 6873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.696007 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.744620 (+ 48613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:08.745356 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:08.745476 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:08.751159 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:08.751166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:08.751171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:08.751222 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:08.756119 (+ 4897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.757050 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.757057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.757061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.757222 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:08.757759 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:08.757765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:08.757769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:08.758362 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:08.758660 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:08.759263 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 15:50:08.759514 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:50:08.766158 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb1d8f40d30422f8406984fcdbbebfb. Latest consensus state: current_term: 1 leader_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.766510 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:08.769714 4015 mvcc.cc:205] Tried to move back new op lower bound from 7334341053307494400 to 7334341053051482112. Current Snapshot: MvccSnapshot[applied={T|T < 7334341053307494400}]
I20260928 15:50:08.773216 3897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865: No bootstrap required, opened a new log
I20260928 15:50:08.787102 3986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.787504 3986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:08.791126 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.791383 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:08.794199 3897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } }
I20260928 15:50:08.796491 4008 catalog_manager.cc:1347] Generated new cluster ID: 3c751c0994a54f1cbb42a0a4f3fd32cf
I20260928 15:50:08.796593 4008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:08.812502 3897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab001d6f87d48fd9fe121a5cca33865, State: Initialized, Role: FOLLOWER
I20260928 15:50:08.813221 3897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } }
I20260928 15:50:08.822640 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.825196 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:08.825054 3897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:08.829303 3870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:08.829566 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ab001d6f87d48fd9fe121a5cca33865"
I20260928 15:50:08.834231 3870 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [term 1 FOLLOWER]: Refusing update from remote peer 6cb1d8f40d30422f8406984fcdbbebfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:08.842314 3871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cb1d8f40d30422f8406984fcdbbebfb in term 1.
I20260928 15:50:08.844076 3986 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:08.859911 4008 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:08.861811 4008 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:08.894548 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb1d8f40d30422f8406984fcdbbebfb. Latest consensus state: current_term: 1 leader_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.894913 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:08.895493 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.895963 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:50:08.897305 3986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb1d8f40d30422f8406984fcdbbebfb" member_type: VOTER last_known_addr { host: "127.13.5.126" port: 39485 } } peers { permanent_uuid: "1ab001d6f87d48fd9fe121a5cca33865" member_type: VOTER last_known_addr { host: "127.13.5.125" port: 41727 } } peers { permanent_uuid: "85323e5f198d4a24b85b90d15a0c928b" member_type: VOTER last_known_addr { host: "127.13.5.124" port: 33271 } } }
I20260928 15:50:08.897694 3986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:08.921137 4008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb1d8f40d30422f8406984fcdbbebfb: Generated new TSK 0
I20260928 15:50:08.922943 4008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:50:08.945125 4040 catalog_manager.cc:1259] Loaded cluster ID: 3c751c0994a54f1cbb42a0a4f3fd32cf
I20260928 15:50:08.945307 4040 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865: loading cluster ID for follower catalog manager: success
I20260928 15:50:08.949739 4040 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865: acquiring CA information for follower catalog manager: success
I20260928 15:50:08.952668 4040 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ab001d6f87d48fd9fe121a5cca33865: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 15:50:09.074007 3976 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.047565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.13.5.126:39485)
0928 15:50:09.047782 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.047791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.070248 (+ 22457us) server_negotiation.cc:407] Connection header received
0928 15:50:09.072205 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.072214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.072319 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.072404 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.073737 (+ 1333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41720: BlockingRecv error: recv got EOF from 127.0.0.1:41720 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 15:50:09.077409 3902 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.053643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.13.5.125:41727)
0928 15:50:09.053865 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.053871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.075298 (+ 21427us) server_negotiation.cc:407] Connection header received
0928 15:50:09.075470 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.075476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.075598 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.075709 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.077165 (+ 1456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58186: BlockingRecv error: recv got EOF from 127.0.0.1:58186 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 15:50:09.079667 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.065667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.13.5.124:33271)
0928 15:50:09.065883 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.065889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.075998 (+ 10109us) server_negotiation.cc:407] Connection header received
0928 15:50:09.078381 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.078391 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.078492 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.078568 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.079440 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58100: BlockingRecv error: recv got EOF from 127.0.0.1:58100 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 15:50:09.137844 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.121257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.13.5.125:41727)
0928 15:50:09.121481 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.121488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.128266 (+ 6778us) server_negotiation.cc:407] Connection header received
0928 15:50:09.129862 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.129870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.130099 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.130253 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.131031 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.132515 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.134555 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.135253 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.135359 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.136170 (+ 811us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.136945 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.137080 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.137333 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.137450 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 15:50:09.206735 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.189672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.13.5.124:33271)
0928 15:50:09.189903 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.189913 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.204528 (+ 14615us) server_negotiation.cc:407] Connection header received
0928 15:50:09.204866 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.204873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.205030 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.205122 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.206432 (+ 1310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58104: BlockingRecv error: recv got EOF from 127.0.0.1:58104 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:50:09.267793 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.251019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.13.5.125:41727)
0928 15:50:09.251272 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.251282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.260521 (+ 9239us) server_negotiation.cc:407] Connection header received
0928 15:50:09.261129 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.261136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.261262 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.261344 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.262143 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.263286 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.265392 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.266112 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.266219 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.266876 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.267049 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.267113 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.267347 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.267441 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 15:50:09.311350 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.302284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.13.5.124:33271)
0928 15:50:09.302484 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.302491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.309347 (+ 6856us) server_negotiation.cc:407] Connection header received
0928 15:50:09.309853 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.309859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.309954 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.310037 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.311084 (+ 1047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58118: BlockingRecv error: recv got EOF from 127.0.0.1:58118 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:50:09.357334 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.350023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.13.5.125:41727)
0928 15:50:09.350235 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.350242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.350792 (+ 550us) server_negotiation.cc:407] Connection header received
0928 15:50:09.351006 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.351013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.351113 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.351243 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.351760 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.353000 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.355119 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.355832 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.355994 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.356484 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.356645 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.356744 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.356897 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.356993 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 15:50:09.367034 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.364759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.13.5.124:33271)
0928 15:50:09.364976 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.364982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.365309 (+ 327us) server_negotiation.cc:407] Connection header received
0928 15:50:09.365653 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.365658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.365747 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.365851 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.366775 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58120: BlockingRecv error: recv got EOF from 127.0.0.1:58120 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:50:09.381451 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.375025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.13.5.125:41727)
0928 15:50:09.375283 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.375289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.375605 (+ 316us) server_negotiation.cc:407] Connection header received
0928 15:50:09.375754 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.375760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.375852 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.375991 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.376359 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.377629 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.379321 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.380136 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.380299 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.380670 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.380820 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.380927 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.381046 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.381131 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260928 15:50:09.390439 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.388312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.13.5.124:33271)
0928 15:50:09.388633 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.388639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.388911 (+ 272us) server_negotiation.cc:407] Connection header received
0928 15:50:09.389129 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.389133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.389209 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.389293 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.390199 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58130: BlockingRecv error: recv got EOF from 127.0.0.1:58130 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
I20260928 15:50:09.409121 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.402664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.13.5.125:41727)
0928 15:50:09.402891 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.402897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.403287 (+ 390us) server_negotiation.cc:407] Connection header received
0928 15:50:09.403504 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.403511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.403626 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.403742 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.404288 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.405424 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.406925 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.407615 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.407746 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.408186 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.408365 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.408474 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.408610 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.408690 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 15:50:09.417871 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.415933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.13.5.124:33271)
0928 15:50:09.416378 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.416385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.416481 (+ 96us) server_negotiation.cc:407] Connection header received
0928 15:50:09.416700 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.416706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.416805 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.416916 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.417623 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58136: BlockingRecv error: recv got EOF from 127.0.0.1:58136 (error 108)
Metrics: {"server-negotiator.queue_time_us":299}
I20260928 15:50:09.445240 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.437585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.13.5.125:41727)
0928 15:50:09.437776 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.437782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.438338 (+ 556us) server_negotiation.cc:407] Connection header received
0928 15:50:09.438538 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.438546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.438648 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.438761 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.439193 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.440443 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.442577 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.443287 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.443440 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.444427 (+ 987us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.444580 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.444689 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.444852 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.444960 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 15:50:09.459699 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.457351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.13.5.124:33271)
0928 15:50:09.457534 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.457542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.458051 (+ 509us) server_negotiation.cc:407] Connection header received
0928 15:50:09.458234 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.458241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.458370 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.458488 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.459470 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58146: BlockingRecv error: recv got EOF from 127.0.0.1:58146 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:50:09.500661 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.495091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.13.5.125:41727)
0928 15:50:09.495401 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.495408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.495719 (+ 311us) server_negotiation.cc:407] Connection header received
0928 15:50:09.495969 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.495974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.496088 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.496170 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.496693 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.497404 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.498986 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.499441 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.499536 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.499942 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.500043 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.500114 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.500246 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.500310 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 15:50:09.509287 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.507163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.13.5.124:33271)
0928 15:50:09.507432 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.507439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.507719 (+ 280us) server_negotiation.cc:407] Connection header received
0928 15:50:09.508018 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.508023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.508122 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.508235 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.509108 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58162: BlockingRecv error: recv got EOF from 127.0.0.1:58162 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 15:50:09.582916 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.577428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.13.5.125:41727)
0928 15:50:09.577671 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.577678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.578018 (+ 340us) server_negotiation.cc:407] Connection header received
0928 15:50:09.578270 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.578276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.578372 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.578479 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.578958 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.579720 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.581128 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.581593 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.581731 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.582172 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.582284 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.582388 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.582548 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.582636 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 15:50:09.590257 3975 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:09.588451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.13.5.124:33271)
0928 15:50:09.588689 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.588696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.588892 (+ 196us) server_negotiation.cc:407] Connection header received
0928 15:50:09.589045 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.589048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.589145 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.589232 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:09.590000 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58178: BlockingRecv error: recv got EOF from 127.0.0.1:58178 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 15:50:09.641434 3995 catalog_manager.cc:1259] Loaded cluster ID: 3c751c0994a54f1cbb42a0a4f3fd32cf
I20260928 15:50:09.641538 3995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: loading cluster ID for follower catalog manager: success
I20260928 15:50:09.644263 3995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: acquiring CA information for follower catalog manager: success
I20260928 15:50:09.646178 3995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85323e5f198d4a24b85b90d15a0c928b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:50:09.727705 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.722033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.13.5.125:41727)
0928 15:50:09.722291 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.722299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.722624 (+ 325us) server_negotiation.cc:407] Connection header received
0928 15:50:09.722885 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.722890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.722964 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.723079 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.723530 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.724273 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.725729 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.726171 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.726314 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.726794 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.726904 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.727007 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.727239 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.727342 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:50:09.740219 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.734017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.13.5.124:33271)
0928 15:50:09.734216 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.734220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.734577 (+ 357us) server_negotiation.cc:407] Connection header received
0928 15:50:09.734789 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.734795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.735006 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.735134 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.735619 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.736543 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.737953 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.738378 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.738535 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.738958 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.739603 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.739720 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.739813 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.739975 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:50:09.756289 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.746796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.13.5.124:33271)
0928 15:50:09.747018 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.747027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.747802 (+ 775us) server_negotiation.cc:407] Connection header received
0928 15:50:09.748196 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.748204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.748328 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.748407 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.749396 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.750892 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.753808 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.754441 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.754569 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.755450 (+ 881us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.755560 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.755633 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.755888 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.755976 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:50:09.762030 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.746838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.13.5.125:41727)
0928 15:50:09.747078 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.747086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.755322 (+ 8236us) server_negotiation.cc:407] Connection header received
0928 15:50:09.755761 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.755767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.755853 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.755983 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.756766 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.757810 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.759969 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.760659 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.760766 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.761247 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.761394 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.761489 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.761621 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.761722 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 15:50:09.762285 4149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:09.747174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.13.5.126:39485)
0928 15:50:09.753959 (+ 6785us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:09.753966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:09.753984 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:50:09.754113 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:09.754118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:09.754351 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:09.754593 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:09.755420 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.756623 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.759627 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:09.760303 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:09.760570 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:09.761082 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:09.761637 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:09.761741 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:09.761891 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 15:50:09.761977 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6647,"thread_start_us":6535,"threads_started":1}
I20260928 15:50:09.767452 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3754
I20260928 15:50:09.768714 3816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:09.882650 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3754
W20260928 15:50:09.905004 3847 connection.cc:535] server connection from 127.0.0.1:41848 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:50:09.905473 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3828
I20260928 15:50:09.906843 3891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:10.018551 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3828
I20260928 15:50:10.040334 13333 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu pid 3905
I20260928 15:50:10.041644 3967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:10.154814 13333 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Pa7uo/build/asan/bin/kudu with pid 3905
2026-09-28T15:50:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4595 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9642 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (371443 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST